Interaction with other IT projects: EUDAT2020, VLDATA, ENVRI PLUS,

Transcription

1 Interaction with other IT projects: EUDAT2020, VLDATA, ENVRI PLUS, A. Spinuso, L. Trani, A. Strollo and D. Bailo EPOS PP final meeting, Rome, October 2014

2 OUTLINE WG1 and the EIDA use case A modular and integrated development plan Intra- and inter- community accvices EUDAT2020 VLDATA ENVRI PLUS Conclusions

3 WG1 Seismology and EIDA EIDA is a distributed data centre established to securely archive seismic waveform + metadata, gathered by European research infrastructures, and provide transparent access to the archives by the geosciences research communices. hqp:// eu.org/eida/eida.html

4 WG1 Seismology and EIDA EIDA is a distributed data centre established to securely archive seismic waveform + metadata, gathered by European research infrastructures, and provide transparent access to the archives by the geosciences research communices. hqp:// eu.org/eida/eida.html

5 WG1 Seismology and EIDA EIDA started to operate formally under ORFEUS in 2013 with two bodies: the management board and the technical commission. EMB and ETC concnually addresses challenges such as efficient data management, content metadata, quality maintenance, provenance and access services in a distributed network of archives. WG7 and others

6 The EIDA NG development plan modular and integrated

7 Intra- community accvices Discussing, proposing and implemencng new standards within the Seismological community: The recent joint effort (EIDA- IRIS DMC) to define the usage of DOIs for seismic networks. The FDSN through its WG III on Products, Tools, and Services has agreed upon a method to provide a>ribu@on to permanent and temporary seismic networks using FDSN Network Codes hqp:// DOIFDSN.pdf The discussion concnues within ORFEUS and EIDA about possible extension of fdsnws and proposals for new ws as standards for FDSN.

8 Inter- community accvices AAI, federated idencty management: Recently ORFEUS and EIDA started the discussion with GEANT/eduGAIN about possible synergies. The EIDA developers are currently tescng possible solucons supported by the edugain experts. A specific workshop is being organized (GEANT- ORFEUS- EIDA) to discuss the E- Infrastructure requirements of users, projects, collaboracons in the broad field of Earth Sciences and discuss how to co- ordinate the support given to users by E- Infrastructures. A specific discipline being targeted is seismology and the workshop is open to other disciplines. hqp://

9 Inter- community accvices Workshop on E- Infrastructure support for Earth Sciences Amsterdam, January 2015 E- Infrastructure providers (GEANT, EGI, PRACE, EUDAT), Research Infrastructures (EIDA and EPOS) along with a small but representacve number of users and projects will meet to discuss the collaboracon models of the projects in Earth Sciences and define how best the e- Infrastructures can support them. From the perspeccve of the e- Infrastructures, topics which may be discussed in detail at the workshop include: ConnecCvity Services and performance monitoring Federated IdenCty Management (AAI/ edugain) compucng needs and types of compucng (HTC, HPC, GPPGU, etc.) cloud provisioning of services: cloud compucng and cloud storage, on- demand provisioning virtual research environments user support model Contact Luca and Alessandro if you would like to parccipate. To make the workshop effeccve the organizers would like to keep the number restricted (1-2 representacves max for each EPOS TCS)

10 Project starts INGV among the partners The EIDA and EUDAT interaccons Cmeline EIDA proposal submiged (led by INGV, KNMI, GFZ) EUDAT 2020 Proposal submiged EUDAT Project ends Pilot projects with communi2es Call for collabora2on EUDAT2020 proposal Oct 2011 Jan 2013 Jan 2014 Sep 2014 Oct 2014 KNMI/Orfeus SURFsara EIDA proposal accepted

11 Overview current B2 services

12 Moving forward with EUDAT Consolidate and improve the CDI together with the scientific communities - Provide full research data lifecycle support - New services for data curation, provenance and workspaces - Workflows and semantic web services - Strengthen interoperability and collaboration with other e-infrastructure providers (eg.:prace, EGI,GEANT) And much more

13 EIDA/EPOS- S plans within EUDAT Set up a robust, scalable, reliable and secure infrastructure to support the federation of datacenters - Enrich data streams with PIDs and detailed metadata - Enable reproducibility and provenance (continue work done with DOI) - Automatic safe replication of datasets on external data resources - Effective exploitation of replicas: reliability, failover, disaster recovery, computation, optimisation - Federated Discovery and Access - Federated Identity Management (in discussion with edugain/geant)

14 ETH/SED GFZ ODC INGV IPGP

15 ETH/SED GFZ ODC INGV IPGP

16 ETH/SED GFZ ODC INGV IPGP

17 ETH/SED INGV GFZ ODC IPGP

18 B2?! Data Discovery Data Access Data ComputaCon AAI Citation Workflows ETH/SED GFZ B2?! New node B2?! INGV ODC B2?! IPGP

19

20

21 What Data VERCE handles: SyntheCc seismograms, plots, 3D Geometry, Videos, KMZ packages, meshes and models. (100 stacons = 900 products and metadata ) 6-10 GB for a single user run, on 1000 cores in ~ 1 hour. Data volume will increase with the acquisicon of observaconal data from EIDA, for the Misfit CalculaCon workflow. (June 2015)

22 VLDATA ObjecCves Create a secure, open and generic planorm - Efficient and cost- effeccve solu2ons for handling large- scale distributed data and heterogeneous infrastructures (EUDAT, EGI, PRACE ) Integrate and advance state- of- the- art e- Science technologies - Integrate DIRAC and SCI- BUS and iden2fy cri2cal APIs helping users to unlock high scien2fic value from large data resources Support a new generacon of data sciencsts - Co- design approach driven and validated by the user community. Ensure the sustainability - VLDATA will evolve around a reference model, demonstra2ng its effeccveness in enabling new sciencfic discoveries

23 VLDATA Cyber Infrastructure

24 VLDATA and EPOS Thirteen world- class large- scale sciencfic research infrastructures parccipate in VLDATA: Astrophysics (GAIA, STARNET, LasMOG), Belle II, BESIII, CTA, DRIHM, EISCAT- 3D, IceCube, LHCb, MolSim (MosGrid and CMMST), NA62, PAO and EPOS. EPOS gain from VLDATA : - Orchestrate and manage services provided by other ini2a2ves ( EUDAT, EGI, PRACE ). - Workflow management and enactment. - VisualizaCon services. - ValidaCon procedures via provenance exploracon. - Discovery services across catalogues (such as the EPOS catalogue) of observa2onal and experimental data..

25 ENVRI+ Goal Provide common solucons to shared challenges for European Environmental and Earth System Research Infrastructures (RIs) in their efforts to deliver new services for science and society.

26 Basic concept for environmental cross- cudng services and solu2ons Core work packages: disciplines Integra2on and coordina2on Users/ data services Users / services Users / modelling Solid Earth Biological ecosystems Oceans Atmosphere Integra2on work packages: user oriented Outreach (internal + external) / dissemina2on

27 ENVRI+ Themes (WPs) 1. Sensors (new sensors, harmonize technologies) 2. Common solu2ons for data (discovery, use, access, workflows ) 3. Common Policies for access 4. Interac2ons between RIs and societ 5. Knowledge transfer and training best prac2ces 6. RI coopera2on framework

28 ENVRI+ Themes (WPs) 1. Sensors (new sensors, harmonize technologies) 2. Common solucons for data (discovery, use, access, workflows ) 3. Common Policies for access 4. Interac2ons between RIs and societ 5. Knowledge transfer and training best prac2ces 6. RI coopera2on framework

29 ENVRI+ & EPOS 2.1 Reference model guided RI design Reference model, ontological framework, design and implementa2on plan 2.3 Data processing and analysis efficiency of data processing, performance of the research infrastructures 2.4 Reference model guided RI design data cura2on, catalogue interoperability, data provenance and tracing 2.5 Service valida2on and deployment

30 ENVRI+ & EPOS

31 EGI Competence Centers BEFORE: EPOS archives provide high quality Earth Science data across Europe, but there is no federated compu2ng infrastructure suppor2ng such analysis. AFTER: adop2on of AAI and deployment of services to support the EPOS community at IaaS, PaaS and SaaS level, demonstra2ng to fulfil the requirements of the researchers, enabling an efficient data analysis transparently using distributed and federated resources

32 EGI Competence Centers Contribu2on in EPOS ICS- d processing modules Results and requirements to be discussed between ICS- C and TCS Processing resources procurement (open issue)

33 PotenCal contribucon to EPOS (TCS, ICS) EUDAT2020: focus on data cura2on services, via Common Data Services to various research communi2es, towards a Collabora2ve Data Infrastructure. e.g. provide data mangement and PID to TCS nodes VLDATA: gathers world- class exper2se to inves2gate interoperable access to heterogeneous resources, flexible workflows and scien2fic gateways. e.g. common interface to some ICS- d ENVRI+: catalogue interoperability, data provenance and tracing, other data solu2ons e.g. help EPOS ICS- C in producing interoperable metadata catalogue EGI CC: enabling an efficient data analysis transparently using distributed and federated resources, AAAI solu2ons e.g. help EPOS ICS- C TCS in AAAI implementa@on