Workspaces Concept and functional aspects

Transcription

1 Mitglied der Helmholtz-Gemeinschaft Workspaces Concept and functional aspects A You-tube for science inspired by the High Level Expert Group Report on Scientific Data Morris Riedel, Peter Wittenburg, Daan Broeder, Bernd Schuller et al.

2 Outline Science Influencing Factors in Context YouTube-like Concept Vision Some Functional Aspects Summary & Conclusions 2

3 Some Talk Goals Separating hype from reality are there realistic core building blocks? Possibilities of a YouTube for science away from explorer thinking List functional aspects that might be relevant and also realizable Taking boundary conditions (heterogeneous scientific landscape, community & data center resources, technologies, etc.) into account Integrate lessons learned from existing research infrastructures around computing (i.e. within HPC and HTC-driven infrastructures) Product-agnostic aim to not mention any product (no marketing) 3

4 Science Influencing Factors in Context 4

5 High Level Expert Group Report on Scientific Data Existing data users in data/community center today in 2010 Current EU calls addressing data needs towards 2020 The report mentions issues & challenges towards 2030 Researchers and practitioners from any discipline are able to find, access and process the data they need. They can be confident in their ability to use and understand data, and they can evaluate the degree to which that data can be trusted. 5

6 ESFRI List of Projects & Roadmap ESFRI = European Strategy Forum on Research Infrastructures A huge set of user communities requiring data & compute infrastructures Resources for geographically dispersed user communities 6

7 ESFRI Projects Number / Research Area Statistics = 44 projects (~ 20 B in ~ 10 years) out of ~ 240 proposals 7

8 ESFRI Projects Evolution / Research Area Statistics Update: 10 more projects 8

9 ESFRI Projects Biological and Medical Sciences 9

10 ESFRI Projects Environmental Sciences 10

11 ESFRI Projects Social Sciences and Humanities 11

12 Experience from (Computing)-Infrastructures Partly within community centers of ESFRI Within ESFRI projects several IT-based infrastructures exist Partnership for Advanced Computing in Europe (PRACE) Also an ESFRI project but of a special kind Other computing-focused e-infrastructures Distributed European Infrastructure for Supercomputing Applications (DEISA) European Grid Infrastructure (EGI) 12

13 Increased Web Community Building & Technologies 13

14 YouTube-like Concept Vision 14

15 Three-layer Foundation Community Virtual Research Environments Virtual Workspaces Community center Storage & Compute Resources and Measurement Devices Data center Storage & Compute Resources 15

16 Basic Overall Concept Web 2.0 YouTube-like Features Scientific Application-specific VRE Workbench Features & Profile Access Other VREs Community Virtual Research Environments authentication AppServer Core Functions End-user Quota User Access Policies Archives/Repositories List of Profiles Community Time authorization Compute resource Service Adapters Configured Filters Devices Configured Filters Virtual Workspaces & Chosen Profile From many Available Preconfigured User Profiles Community center Storage & Compute Resources and Measurement Devices Storage Compute resource Generic and Common Services Data center Storage & Compute Resources 16

17 Work Spaces Core Building Blocks Core Functions Dynamic Web 2.0 elements, mash-ups, glue between other elements, etc. List of profiles Workspaces are bound to profiles of a scientific end-user (e.g. role dependent) Service Adapters Data & computing services, workflow services, GreenIT services, cloud access Policies, Quota, User/Community Time Computational time for a community, quotas of storages, end-user policies Filter (i.e. pre-configured search/setup) End-user Quota Core Functions Service Adapters Filter for available services (e.g. from 400 to 12 in context) List of Profiles User Access Policies Filter for data (e.g. publications last year, measurements in 2010, ) Community Time Configured Filters 17

18 Some Functional Aspects 18

19 Handling Data-types Functional Aspect Existing major (community) data types / viewers Standards (e.g. within ISO) exists and are used in data viewers NetCDF, XML, RDF, RSS/Atom, etc. Metadata is as crucial as persistent IDs Other specific data types / viewers Create own data types and share it with scientific community Provide functionality to share this, also the data viewer of this data-type (Service-based) registries for data-types Data duplication warnings? Similar data existing? Community acceptance and uptake of (new) data-type Short-term: collaborations now, then rating / voting Med-term: use by other scientists and used in comparisons Long-term: perhaps open standard in a relevant standard body... Create non-specialist as well as specialist data access, visualization, mining and research environments. Service Adapters 19

20 Annotation, Tagging and Searching Annotations & Tagging of scientific data Scientists (must) provide these pieces of information to maintain usefulness E.g. when uploading scientific results into a work space E.g. after a computation or data service invocation when new data is created Basically any time when working with data and getting new insights Other scientists should be allowed to tag as well (but that brings us to trust) Search for data Based on Tags, metadata, Possible technical realization Tagging/annotation is possible via data services Whole trusted communities or other scientists can be allowed to tag data (security services) Service Adapters Create annotation services to collect views and derived results. 20

21 Data Recommendation Systems Multiple ways what could make sense Recommendations like scientists that worked on this data also worked on this data Analysis of diversity in US Recommendation like Most viewed data today (e.g. in the CLARIN community) Scientific data is shown together with publications or device data bottom line: all is data, no difference Possible technical realization Data :1980 compared 1990 MPI toolset usage Paper about voice Device #46 data High availability/scalable services that can be realized as data gateways to numerous storages/archives/repositories of different kinds Use data recommender systems i.e. you may also be interested in... 21

22 Functional Aspect of Quotas, Filters & Policies Filters can be used in many contexts E.g. for filtering storages with very high trusts E.g. for filtering a dedicated set of services E.g. for filtering environments (e.g. myexperiment site) Quotas or computational time (on computing resources) Seems to be not hiding the data infrastructure E.g. user have to know how much capacity is there End-user Quota Community Time End-user Time Available Resources (Filter #443): STORAGE: (JUELICH) STORAGE: (RZG) MYEXPERIMENT: (UoS) Configured Filters HPC: JUGENE (JUELICH) 19,78 TB free HPC: (SARA) Trust Claim: HTC: (EGI) RZG gives 50 years guarantees for data. STORAGE: (JUELICH) STORAGE: (RZG) Policies (closely linked with profiles): Security & Trust Partly visible for end-users, but mostly managed by community managers User Access Policies List of Profiles deal with the various filters that different disciplines use when choosing and describing data 22

23 Functional Community Aspects Comments on scientific data contributions create new data or confusion? Disagreements about data (cf. some Wikipedia articles) e.g. interesting to link to an interesting publication in context e.g. enable citizen scientists to be part of the data research studies Prof. Dr. Known Correct. I used this great data in a recent publication supporting this claim, you can see it here: PaperBerlinTraffic Citizen from Berlin I heard that you have used my data from Berlin to claim that the traffic evolution in this region is getting higher in 10 years. Increasing trust in data: number of worked with data / views / ratings Trust (or user reputation) is important in data Improved via supported publications, but also ratings Must be balanced according w.r.t. voting entity (e.g. citizen scientist vs. expert in the field) Trust also increased by the number of views or usage a scientific e-infrastructure that supports seamless access, use, re-use, and trust of data. In a sense, the physical and technical infrastructure 23 becomes invisible and the data themselves become the infrastructure.

24 Real-time Functional Aspect Sensors from ESFRIs Also equipped with PIDs In some ESFRIs real-time information matter Perhaps remote switch-on/off or even steering Show status with sensors or other resource problems E.g. Specific sensors off-line, supercomputer in maintenance, storage robot problems, etc. Questions of level of visibility Should scientists really see the resources? Some prefer certain HPC machines for instance for specific code tuning while HTC users don t care Realization via (sensor) data aggregation Information systems in DEISA and EGI do similar things sensor ship #4455 sensor airplane #4711 HPC: (SARA) HPC: (JUELICH) HTC: (EGI) STORAGE: (JUELICH) STORAGE: (RZG) Wrapping of existing sensor information provider or evolution necessary global scientific asset with all its sensors, instruments, workstations, and networks are truly massive. 24

25 Functional Aspects about Data and Services Encourage scientists to think more about data Not all (unused) data can be removed, especially in long-term preservation Many unused data still exists that would be never used again Links with several other resources (long-term) Video-/tele-conferencing tools, see appointments Meetings and their minutes/recordings are data as well Can be directly put in the work space in context List of several services (balance with usability) Many services should be not seen by end-users Others might be of specific interest Community-specific data-mining algorithms Warning STORAGE: (RZG) 76% of your data in this storage is unused since 90 days remove data? Available Services (Filter #43): Data-mining service Workflow service Workflow services for enhanced data transformations/processings 10:46 - CEST Meetings Today 9:00 Steering Group, see Meeting Minutes A variety of access and curation services that will vary between scientific disciplines and over time. 25

26 Commercial Hooks Many scientific packages are free, but many of them not Different licenses exist: per center / per end-user License server must be accessible via Work spaces Application hooks for scientific packages in Work spaces Towards infrastructure sustainability and creating ROI for business Possible app-store for science - an open market HPC: (SARA) HPC: (JUELICH) STORAGE: (AZURE MS) STORAGE: (AMAZON S3) STORAGE: (RZG) Example from the biological domain, molecular docking software Autodock is a freely available scientific package for molecular docking E.g. Commercial FlexX package as commercial app in work space Potential danger: only users with FlexX app can see/use data results/viewer Create a scientific Davos meeting to bring commercial and scientific domains together. 26

27 Summary & Conclusions 27

28 Summary Warning STORAGE: (RZG) 76% of your data in this storage is unused since 90 days remove data? Available Services (Filter #43): Data-mining service Workflow service 10:46 - CEST Meetings Today 9:00 Steering Group, see Meeting Minutes Available Resources (Filter #443): STORAGE: (JUELICH) STORAGE: (RZG) sensor ship #4455 sensor airplane #4711 HPC: (SARA) HPC: (JUELICH) HTC: (EGI) STORAGE: (JUELICH) STORAGE: (RZG) Prof. Dr. Known Citizen from Berlin Correct. I used this great data in a recent publication supporting this claim, you can see it here: PaperBerlinTraffic I heard that you have used my data from Berlin to claim that the traffic evolution in this region is MPI toolset usage Paper about aging Device #46 data 28

29 Conclusions Physical and technical infrastructure invisible, a data infrastructure only Basic concept addresses this and the three-level architectural model At the same time overview of quotas, policies & data must be preserved Complex: 80/20 rule everywhere! Integration of a wide variety of service/technologies (interoperability!) Many protocols already widely used, e.g. OAI-MPH or Web services based on SOAP or REST, URI-based (open) resources, XML, etc. Security setups diverse but are key aspect of success (rather new is the homeless or citizen scientists that needs to be addressed) Trust in/scaling of work spaces would be crucial and non-trivial, but can be achieved with clustering of services (fail-over strategies, etc.) 29