SHIWA is supported by the FP7 Capacities Programme under contract No. RI

Transcription

1 SHaring Interoperable s for Large- Scale Scientific Simulations on Available s Interoperability Enables Interdisciplinary Research Gabor Terstyanszky, University of Westminster 02 nd February 2012 SHIWA is supported by the FP7 Capacities Programme under contract No. RI

2 Interoperability Challenge

3 Partners: SHIWA Project Computer and Automation Research Institute, Hungarian Academy of Sciences MTA SZTAKI Hungary University of Innsbruck UIBK Austria Charité - Universitätsmedizin Berlin C-UB Germany French National Centre for Scientific Research CNRS France University of Westminster UOW United Kingdom Cardiff University CU United Kingdom Academic Medical Centre of the University of Amsterdam AMC Netherlands University of Southern California USC USA Duration: July 2010 June 2012 (extended to September 2012) 3

4 SHIWA Objectives and Services Objectives: To create an environment which enables seamless execution of workflows of different workflow systems through workflow interoperability Services: supports the whole workflow lifecycle: editing, uploading, browsing downloading and executing workflows provides coarse- and fine-grained workflow interoperability solution offers Distributed Computing Infrastructure interoperability solution provides desktop computer and portal interfaces to manage workflows Key actors: researchers workflow engine developers workflow developers 4

5 WF Developer Upload Developer Portal SHIWA Solution Scien1st Browse Run End User Portal WF system Developer Upload Admin Portal fmri Sequencing MRI simulator NonLinSyst simulator. SHIWA WF Repository WF descrip+on WF executables Master CGI WF System Embedded Run Embedded WF Embedded System 2 a WF System 1 Run WF System 1 Run Run c b b SHIWA WF enactor Target FGI WF System Impor Source t Source WF System IWIR WF System Export Export IWIR IWIR IWIR SHIWA WF system Repository WF engine binary IWIR front,back WF end engine XML description Triana MOTEUR Pegasus ASKALON P-GRADE 5 SHIWA Science Gateway

6 SHIWA Data Structure descrip+on plain text which describes the inputs and outputs and explains what the workflow does implementa+ons It contains all implementa+on files or references to these files (via eg. URLs) and also holds other data/ metadata necessary to run the workflow on its associated workflow engine. configura+ons It contains pre- defined input files and parameters or references to these files and parameters and other data/ metadata of a workflow. engines It contains files and other data/metadata necessary to execute a workflow engine on a grid site or references to them 6

7 s of Engine A Coarse-Grained Interoperability of Engine B J1 J2 WF4 client Service Engine B J3 Engine A native workflows: J1, J2, J3 non-native workflows: WF4 - black boxes which are managed as legacy code applications 7

8 SHIWA Portal admin WS-PGRADE editor SHIWA Science Gateway SHIWA Repository WF1 Repository WF1 1 CGI Infrastructure WFn WFm p local cluster Triana Taverna SHIWA VO glite Globus WS-PGRADE engine with GIB Service Proxy Server SHIWA Proxy Server MOTEUR pre-deployed- s SHIWA Science Gateway Resources na+ve PGRADE local resources: invoca+on of locally deployed s portal PGRADE v2.4.1 submission to local cluster Kepler ASKALON MOTEUR ASKALON GS repository + SHIWA repo remote resources: through remotely pre- deployed submiuer with GIB s to glite and Globus s proxy management SHIWA Proxy Server 8

9 SHIWA Portal: Editor 9

10 SHIWA Portal: Configuring 10

11 SHIWA Portal: Executing 11

12 SHIWA Repository Browse s Browse Implementations Advanced Search Search Sort 12

13 SHIWA Repository 13

14 CGI User Scenario: Native SHIWA VO SHIWA Science Gateway SHIWA Portal SHIWA Repository submit s6 local cluster glite Globus admin WF1 WFn search WF s1 Triana Taverna s3 WF1 WFm MOTEUR PGRADE editor PGRADE engine WF list s4 1 p Repository + WF with GIB s5 Service edit WF s2 researcher s6 Kepler ASKALON invoke MOTEUR pre-deployed- s ASKALON GS Proxy Server SHIWA Proxy Server 14

15 CGI User Scenario: Non-native SHIWA VO SHIWA Science Gateway SHIWA Portal SHIWA Repository local cluster glite Globus admin WF1 WFn Repository search WF s1 Triana Taverna WF1 WFm MOTEUR PGRADE Editor 1 + WF p s5 researcher Kepler ASKALON MOTEUR ASKALON GS PGRADE Engine with GIB Service s4 Client s4 s2 UI MOTEUR editor MOTEUR Engine invoke s3 s6 pre-deployed- s 15

16 Interoperability workflow for A workflow for B J1 J2 J3 Engine JSDL Translator J1 J4 J2 J3 J4 Bridge Metabroker Proxy Server 16

17 Fine-Grained Interoperability of Engine A IWIR workflow of Engine B J1 J1 J1 J2 J3 J2 J3 J2 J3 J4 J4 J4 WFA ->IWIR frontend plug-in WFA ->IWIR backend plug-in Engine B front-end plug-in: converts workflow of workflow engine A into IWIR representation back-end plug-in: converts from IWIR representation into workflow of workflow engine B 17

18 SSP & FGI Infrastructure SHIWA Science Gateway ARC SHIWA Portal SHIWA Repository s Unicore WF1 WFn glite Globus admin Repository WS-PGRADE editor WS-PGRADE engine WF1 1 submitter WFm WFp Bridge GS PGRADE Kepler Triana Taverna JSDL Translator Proxy Server Meta Broker MOTEUR ASKALON pre-deployed- s na+ve ASKALON, MOTEUR, WS- PGRADE local s: Kepler, MOTEUR, Taverna & Triana s portal WS- PGRADE v3.2.2 submit to the local cluster SHIWA Science Gateway Resources repository + SHIWA remote s: ASKALON, GS & MOTEUR s ARC, glite, Globus and Unicore s proxy server

19 Interoperability Scenario SHIWA Science Gateway SHIWA Portal SHIWA Repository WF1 WFn search WF admin Repository WF1 WFm s1 1 WFp s3 WS-PGRADE editor WS-PGRADE engine s4 JSDL Translator s5 edit workflow s2 Bridge submit workflow s7 ARC glite s Unicore Globus s6 Meta Broker s6 Proxy Server 19

20 FGI Interoperability Scenario SHIWA Science Gateway search WF SHIWA Portal SHIWA Repository s1 WF1 WFn admin Repository WF1 1 WFm WFp s4 Editor s2 edit WF WS-PGRADE Editor WS-PGRADE Engine JSDL Translator c Bridge FP1 FP2 FPn s5 Engine s3 s6 c BP1 BP2 BPm Meta Broker s7 ARC Unicore submit WF Proxy Server glite s Globus 20

21 SHIWA Simulation Platform: Access SHIWA Science Gateway (SHIWA Portal + SHIWA Repository) portal and repository account given by the science gateway administrator resources certificate given by the relevant administrator, for example the VO sysadmin Access to the SHIWA Portal shiwa-portal.cpc.wmin.ac.uk/liferay-portal-6.05 Access to the SHIWA Repository shiwa-repo.cpc.wmin.ac.uk 21

22 SHIWA Simulation Platform: Selling Points the simulation platform supports the whole lifecycle of workflows, i.e. creating, testing, uploading, browsing, downloading and running workflows users can use workflows of their own and of other workflow systems through the same user interface, i.e. they can re-cycle or share workflows created by other research teams or even other research communities workflow developers (or e-scientists) can create, run, upload workflows through the simulation platform researchers (astro physicists, earth scientists, bio scientists, etc.) can browse the workflow repository, select and run workflows through the simulation platform the simulation platform provides access to major European e- infrastructure, the European Grid Initiative (EGI) infrastructure which offers large pool of compute and storage resources 22