EUDAT. Towards a pan-european Collaborative Data Infrastructure

Transcription

1 EUDAT Towards a pan-european Collaborative Data Infrastructure Damien Lecarpentier CSC-IT Center for Science, Finland EISCAT User Meeting, Uppsala,6 May 2013

2 2

3 Exponential growth Data trends Zettabytes Exabytes Petabytes Terabytes Gigabytes Increasing complexity and variety Where to store it? How to find it? How to make the most of it? 3

4 Data Curation Trust Collaborative Data Infrastructure -A framework for the future? - Data Generators Users User functionalities, data capture & transfer, virtual research environments Community Support Services Data discovery & navigation, workflow generation, annotation, interpretability Common Data Services Persistent storage, identification, authenticity, workflow execution, mining

5 Data Centers and Communities 5

6 Five research communities on Board EPOS: European Plate Observatory System CLARIN: Common Language Resources and Technology Infrastructure ENES: Service for Climate Modelling in Europe LifeWatch: Biodiversity Data and Observatories VPH: The Virtual Physiological Human All share common challenges: Reference models and architectures Persistent data identifiers Metadata management Distributed data sources Data interoperability 6

7 Communities Data Centers

8 Safe Replication Service Robust, safe and highly available data replication service for small- and medium- sized repositories To guard against data loss in long-term archiving and preservation To optimize access for user from different regions To bring data closer to powerful computers for compute-intensive analysis PIDs Policy rules EUDAT CDI Domain of registered data 8

9 Data Staging Service Support researchers in transferring large data collections from EUDAT storage to HPC facilities Reliable, efficient, and easy-to-use tools to manage data transfers Provide the means to reingest computational results back into the EUDAT infrastructure EUDAT CDI Domain of registered data PRACE HPC HPC 9

10 Simple Store Service Allow registered users to upload long tail data into the EUDAT store Enable sharing objects and collections with other researchers Utilise other EUDAT services to provide reliability and data retention Simple upload Simple metadata PID registration EUDAT CDI Domain of registered data 10

11 Metadata Service Easily find collections of scientific data generated either by various communities or via EUDAT services Access those data collections through the given references in the metadata to the relevant data stores Europeana of scientific data EUDAT CDI Domain of registered data 11

12 Building Blocks of the CDI Metadata Catalogue Aggregated EUDAT metadata domain. Data inventory Data Staging Safe Replication Simple Store Dynamic replication to HPC workspace for processing Data curation and access optimization Researcher data store (simple upload, share and access) AAI Network of trust among authentication and authorization actors

13 EUDAT Sites general data centres community centres representing all the associated community data centres

14 Resource coordination and site monitoring 14

15 New services? Real-time data handling: handling of (near) real time data streams generated by instruments (e.g. earth science) Semantic Annotation: a generic cross-disciplinary solution for improving data sets by establishing references to recognized knowledge sources Crowd Sourcing Web Services Memento Community Interests 15

16 Real Time Data Real time data is every where! Scientific Machines generating data Particle accelerators: LHC Astronomy telescopes: ESO VLT, Lofar, SKA Life sciences: DNA sequencing Different types of sensors: seismology sensors, radar images, gps tracking HPC processing! Other type generated data Web generated data: twitter feeds, weather/traffic websites, stock exchange,.. There is more data on the internet then storage devices to store the data! How to handle this data which is in flux How can data objects or part of data objects be identified 16

17 EPOS s use case data acquisition Seismologists community scenario: Hundreds of stations in Italy, ten of thousands in the world. Real-time acquisition of data as seismological waveforms. 17

18 EPOS s use case data acquisition 5 Arrive between minutes hours days GAP 6 7 GAP 9 Sensor data stream is continuous Data objects are organized by sensor, a file, and time series, data blocks with in a file Sensor data does not always arrive sequentially, but data is recorded sequentially, gaps are created Data objects are replicated (Safe Replication) to other centers When a data object is finished, register with a PID What to do when data objects are updated: replicate again and update PID! Can you reference a separate (part of a) time serie 18

19 EUDAT and EISCAT Several interactions over the last few months EUDAT User forum in London (2013) EISCAT User Meeting, NeiC conference, etc. Interest from EUDAT to interact with EISCAT Mature and organized community requirements easier to identify, possibility to test solutions effectively, sustainability, etc. EISCAT 3D RI new data and computing challenges. Big Data requires strategic changes and new partnerships! EUDAT could also use EISCAT as a use case to develop its services (e.g. Real-Time Data) Several services developed by EUDAT seem to fit EISCAT 19

20 Partnership opportunities EUDAT and EISCAT Storage Long-term or temporary storage on top of EISCAT s capacity Facilitating access to data sets for distributed users Computing Access to HPC machines to perform simulations Real-Time Data How to move forward? Join a pilot now! Free resources and support, exchange of expertise, test your infrastructure, etc. Join the discussion on new services: express requirements, join a working group (e.g. RTD) 20

21 21

22 Welcome to the 2nd EUDAT Conference! Thank you! 22