This vision will be accomplished by targeting 3 Objectives that in time are further split is several lower level sub-objectives:

Transcription

1 Title: Common solution for the (very-)large data challenge Acronym: VLDATA Call: EINFRA-1 (Focus on Topic 5) Deadline: Sep. 2nd 2014 This proposal complements: Title: e-connecting Scientists Call: EINFRA-9 (VRE) Deadline: Jan. 14th Objectives: The mission/vision/endgoal of VLDATA is to provide a common solution for handling large and extremely large common scientific data in a costeffective way. This solution builds on existing pan-european e- Infrastructure and tools to provide an interoperable, efficient and sustainable platform for scientific user communities, in particular, to support a new generation of data scientists. The success of this project will secure European leadership on the development and support of big data and global data science and, therefore, will contribute to the leadership of European scientists and enterprises in many research and innovation fields :) This vision will be accomplished by targeting 3 Objectives that in time are further split is several lower level sub-objectives: O1: a "flexible and extendable" platform supporting common solutions for large scale distributed data processing and analysis, ensuring interoperability among existing e-infrastructure providers. * O1.1: (WP2,3,5) provide a common solution using generic e-infrastructure for processing large scale or extremely large scale of scientific data in a robust, efficient and cost-effective way. * O1.2: (WP6) provide a flexible and customizable platform that can be extended to cover the specific requirements of each community. O2: standardized solutions aiming to a global interoperability of open access for large-scale data processing, minimizing unnecessary large transfers * O2.1: (WP2) provide common language and standard for handling big volume of data * O2.2: (WP2,3,5) improve the efficiency of distributed data processing by providing smart data and computing management platform * O2.3: (WP2,3,5) enable effective handling of big data samples by

2 integrating new technologies * O2.4: (WP8) assess the value of this generic solution towards relevant stakeholders: end scientists, their management, funding agencies, policy makers, companies and the society at large O3: Increase the number of users and Research Infrastructure projects making efficient use of existing e-infrastructure resources, designing appropriate exploitation strategies and a long-term sustainability plan. * O3.1: (WP5,7) deliver ready-to-use high-quality standard products for internal and external usage, enhancing interdisciplinary dat sciences at a global scale * O3.2: (WP6,9) increase the degree of the open access of large scale distributed data * O3.3: (WP9) educate new generation of data scientists and the society in general 1.3 Concept and approach - Concepts (main ideas, models and assumptions): make IT simple * Simplicity: VLDATA provides an abstraction of the different Resources that are all made accessible the end user via the same interfaces. * Transparency: Users are allowed to specify their Workflows/Pipelines with different levels of abstractions. The platform takes care of the necessary Resource Allocation to fulfill the required specifications. * Extendability and flexibility: VLDATA provides an API that allows users to extend the provided functionality by developing new or customized components * Reliability: Quality standards and extensive validation in several scientific domains to ensure the readiness-to-use and robustness of VLDATA based solutions * Scalability: Modular implementation allowing horizontal (amount of connected Resources or Users) and vertical (amount of processed Units) scaling to adapt VLDATA to the needs of each particular community or Research Infrastructure project. * Smart and intelligent: building on collected experience and monitoring

3 data, algorithm can look for optimized scheduling/searching strategies, including automated decision making based on usage traces and expectations. * Cost-effective: Building up on existing well-established solutions and incrementally extending and developing to address new challenges with an evolving validated common solution, avoiding unnecessary duplicated efforts. * Resource description semantic Model (building blocks): - Collaborative modular architecture, with multiple layers sharing the same Framework and Basic modules, allowing horizontal & vertical scaling to ensure scalability. - Open, iterative, incremental and parallel, requirement-driven development process. Agile(?) methodology. - Standard procedures for quality assurance, including security, platform integration and validation, including reference benchmarks, and release procedures in accordance with requirements for production level services. Layers: (result of 10 evolution of DIRAC development effort) - Framework: (communication, security, access control, user/group management, DBs) - Basic modules: SystemLogging, Configuration, Accounting, Monitoring - Low level modules: File Catalog, Resource Status, Request Management, Workload Management - High level modules: Data Management, Workflow Management - Interfaces: User - Resource Assumption: [Describe and explain the overall concept underpinning the project. Describe the main ideas, models or assumptions involved. Identify any trans-disciplinary considerations;] - Current solution can be evolved into the new general platform to be widely applied. - Evolution from grids to clouds, but heterogeneity will increased - Large degree of commonality on low-level requirements and tools between different scientific domains - Fast grow of data and computing requirements almost doubling every year. Aggregated estimation close exabyte level in 5 years from now (EGI expects Cores and 1?? exabyte of scientific data by 2020). (Ref: - Similar grow in number of data objects, computing units and end users (60

4 % of ESFRI projects completed or launched by 2015). - New scientific domains are entering the digital era 4th paradigm of science, new data science is emerging ( collaboration/fourthparadigm/) - Data is to be made openly available beyond the community that produced them, down to the citizens that might also contribute to its further processing - Common development and validation provides robustness as well as costsaving and, thus, enables sustainability - TRL [Describe the positioning of the project e.g. where it is situated in the spectrum from "idea to application", or from "lab to market". Refer to Technology Readiness Levels where relevant.] From 5-6 towards Links [Describe any national or international research and innovation activities which will be linked with the project, especially where the outputs from these will feed into the project;] (Past Inputs) * building on 15 year HEP experience, transferring this expertise and knowledge to other domains * DataGrid -> EGI, WLCG, NorduGrid, OSG, NeCTAR, EUDAT (Concurrent inputs) * EIROForum: (e-infrastructure commons architecture, p, 19) * Forth paradigm of science, (e-infrastructure providers) * EUDAT-2, EGI-2, EduGain, RDA, SWAMP, GEANT, PRACE, NRENs, Commercial Providers (Technology providers) DIRAC dcache InSilicoLab ARC (Clients) * ESFRIs * ESFRI-clusters * other Relevant Research Infrastructure projects (like those in the European strategy for particle physics) at national or international level, that may or may not have yet access to large resources the e-infrastructure * Other data sources: Smart cities, sensor networks, (Competitors, possible future partners)

5 Condor (misses connection Workload - Data) Panda (misses generality) glite (misses community management, too complex) globus-online Related Work Commercial cloud solutions HelixNebula Microsoft Azure GoogleApps Research Community <-> Experiments (Research Infrastructure projects) 1.4 Ambition [Describe the advance your proposal would provide beyond the state-of-theart, and the extent the proposed work is ambitious. Your answer could refer to the ground-breaking nature of the objectives, concepts involved, issues and problems to be addressed, and approaches and methods to be used. Describe the innovation potential which the proposal represents. Where relevant, refer to products and services already available on the market. Please refer to the results of any patent search carried out.] - Methodology G. Technology readiness levels (TRL) Where a topic description refers to a TRL, the following definitions apply, unless otherwise specified: -TRL 1: basic principles observed -TRL 2: technology concept formulated -TRL 3: experimental proof of concept -TRL 4: technology validated in lab -TRL 5: technology validated in relevant environment (industrially relevant environment in the case of key enabling technologies) -TRL 6: technology demonstrated in relevant environment (industrially relevant environment in the case of key enabling technologies) -TRL 7: system prototype demonstration in operational environment -TRL 8: system complete and qualified -TRL 9: actual system proven in operational environment (competitive manufacturing in the case of key enabling technologies; or in space)

6 2. Impact 2.1 Expected impact [Please be specific, and provide only information that applies to the proposal and its objectives. Wherever possible, use quantified indicators and targets.] [The mission/vision/end-goal of VLDATA is to provide a common solution for handling large and extremely large common scientific data in a costeffective way. This solution builds on existing pan-european e- Infrastructure and tools to provide an interoperable, efficient and sustainable platform for scientific user communities, in particular, to support a new generation of data scientists. The success of this project will secure European leadership on the development and support of big data and global data science and, therefore, will contribute to the leadership of European scientists and enterprises in many research and innovation fields :)] DIRECT impact. scalability, robustness (for the Research Infrastructure) The expected impact is that participating RI projects will be able to operate their Distributed Computing Systems efficiently processing their large volume research data, making it available to their end users in reliable and cost-effective way, which couldn't be achieved before, which may lead to new way of organizing science activities, leading to significant scientific break throughs. By providing important functional components (e.g., ) which was missing from existing practices, VLDATA platform will make possible the transparent integration of resources, hiding the complexity from use, resulting in the extension of the scale of the resources Resource Infrastructure projects can utilize. This will increase the number of RI using the project tools and the number of different types of resources reachable through the tools. :) simplicity (for the user: scientist/operator) cost-efficiency (for funding agencies) reduce the duplication efforts, maximizing the use of EU-invest e- Infrastructures, enlarging the user communities, providing efficient data processing services, providing advanced technology by integrating the state-of-the-art which reduces development cost significantly. (also the processing algorithm ) avoid lock-in Indirect impact: large user community

7 - science - innovation - society - industry - citizens - policy maker - new generation data scientists On the other hand, the scale of the data challenge requires simple but intelligent solutions to integrate resources from different e- Infrastructure providers. 2.2 Measures to maximize impact a) Dissemination and exploitation of results [Dissemination and exploitation measures should address the full range of potential users and uses including research, commercial, investment, social, environmental, policy making, setting standards, skills and educational training.] [The approach to innovation should be as comprehensive as possible, and must be tailored to the specific technical, market and organizational issues to be addressed.] b) Communication activities [Describe the proposed communication measures for promoting the project and its findings during the period of the grant. Measures should be proportionate to the scale of the project, with clear objectives. They should be tailored to the needs of various audiences, including groups beyond the project's own community. Where relevant, include measures for public/societal engagement on issues related to the project.] c) Internationalization WP6:

8 Belle II: - Usage of DIRAC for the Experiment, use case presented: * Common access to various platforms: Grid + cloud + cluster + HPC * Support for Monitoring for Workflow management tools * Integrate for the needs of other participants * User interface EU-T0: Virtual data centers / New Virtualization techniques? PAO: - Usage of DIRAC for the Experiment, data taking -> 2022 * using a standard solution will help the sustainability. * Extend functionality for their use case. * Common access to various platforms: Grid + cloud + cluster + HPC (follow evolution of providers) in particular OSG * Open Access to data EU-T0: Data locality LHCb: - should cover Run 2 needs and target to the needs of Run 3 (DAQ Upgrade) * Data rate will be increase by a factor ~5, 10 PB/year. * Integration of Cloud resources. * Massive data-driven Workflows for users. * Data preservation (?) * Resource (cpu/storage/network/...) description/monitoring/availability/ management, smart allocation * Smart/Intelligent/dynamic data placement strategies (network) EU-T0: New Virtualization techniques, Resource description/monitoring/ availability, Virtual data centers, Data locality EISCAT_3D: * searching data (metadata catalog), intelligent searching (patterns recognition) * visualization, * flexible interconnection of different resources, central (HPC) + distributed (Grid/Cloud) * time constrained massive data reduction (10 PB -> 1 PB / month??), including the possibility for users defined algorithm. EU-T0: Others: - EU-T0: - Virtual distributed data centers on demand (STN) - Data locality (Smart/Intelligent/dynamic data placement strategies (STN)) - New Virtualization techniques

9 WP1 Coordination (UB, Spain) External Advisory board (EUDAT, OGF, RDA, OSG, ) WP2 Requirement analysis & Design (CU, UK) WP3 Data-driven development( UB, Spain) WP4 User-driven development( CYFRONET, Poland) WP5 ( UAB, Spain) WP6 XXXXXX - LHCb (CNRS/INFN) - Belle II (Institut Jozef Stefan, UniMB, Mariborand UniLJ,Slovenia) - EISCAT_3D (SNIC, Sweden/EISCAT Science Associate) - PAO (CESNET, Czech Republic) - BES III (INFN-Torino, Italy) - XXXX - YYYY - DIRAC 4 EGI, multi-community solution EGI ( EGI.eu, the Netherlands) WP7 Dissemination (CNRS, France) EGI.eu EUDAT? WP8 Exploitation (ETL, UK) Open DISData Collaboration WP9 Communication, Internationalization (UvA, the Netherlands) DIRAC Consortium, EGI.eu WP 7 -> WP 9 Communication (Internationalization) WP 8 -> Exploitation (business/revenue model) WP 9 -> WP 7 Dissemination (+training) Description Objectives Tasks -> Deliverables Outputs (Internal + External) Risks Possible relevant Milestone

10