Processing big data by WS- PGRADE/gUSE and Data Avenue

Size: px
Start display at page:

Download "Processing big data by WS- PGRADE/gUSE and Data Avenue"

Transcription

1 Processing big data by WS- PGRADE/gUSE and Data Avenue Peter Kacsuk, Zoltan Farkas, Krisztian Karoczkai, Istvan Marton, Akos Hajnal, Tamas Pinter MTA SZTAKI SCI-BUS is supported by the FP7 Capacities Programme under contract nr RI

2 Processing big data by workflows Processing big data many times requires a set of activities that can be combined and formulated in a scientific workflow in order to repeat the activities for a large set of data components in an automatic way. Therefore scientific workflows that can run in Globus-based DCIs and they can access large data storages are crucial for processing big data. 2

3 The SCI-BUS approach Integrate workflows with the Data Avenue services Run these workflows in an environment that enables to run the nodes of a workflow in many different type of DCIs to achieve Highly parallel and distributed workflow execution Workflow level interoperability among DCIs and data storages The environment offered by SCI-BUS is the WS- PGRADE/gUSE gateway framework 3

4 WS-PGRADE/gUSE Generic-purpose gateway framework Based on Liferay General purpose Workflow-oriented gateway framework Supports the development and execution of workflow-based applications Supports the fast development of domainspecific gateways by a customization technology Most important design aspects are flexibility and robustness 4

5 Flexibility in exploiting parallelism Multiple instances of the same workflow with different data files Parallel execution inside a workflow node Parallel execution among workflow nodes Parameter study execution of the workflow Multiple jobs run parallel Each job can be a parallel program

6 Flexibility of using various DCIs Flexible management of Security: Individual users certificate Robot certificates Flexible access to various types of DCIs: Clusters (PBS, LSF, MOAB, SGE) Cluster grids (ARC, glite, GT2, GT4, GT5, UNICORE) Supercomputers (e.g. via UNICORE) Desktop grids (BOINC) Clouds 6

7 Using IGE Globus resources in the DRIHM gateway start.sh $modelname $jobid 7

8 Flexibility in data storage access Use Data Avenue Blacktop service To access data storages in different DCIs To transfer files among the storages of different DCIs To upload/download files to/from the storages of different DCIs Data Avenue Liferay portlet to access the data transfer services of Data Avenue Blacktop See details: Currently supported protocols: http, https, ftp, gsiftp, srm, S3, (irods in beta phase) Soon coming protocols: LFC, further cloud storage protocols 8

9 Data Avenue services Data SZTAKI Data XY Data Avenue Portlet WS-PGRADE gateway Data Avenue Blacktop service Openstack FS1 FS2 FS3 FSn Amazon glite GT5 9

10 Use cases to be supported Browse, download, upload Create dir, Remove item, Data Avenue Produce data Use data Storage Service Storage Service Storage Service EGI Community Forum 2014, Helsinki, Finland 10

11 Data Avenue services Data Avenue Blacktop: Core service accessible through SOAP (Java API provided) Hides access details of storage services Data Avenue Portlet: User-friendly interface to manage data, up-, and download files,... Can be deployed onto any Liferay-based portal Data Avenue in WS-PGRADE/gUSE: Integration in a science gateway enabling easy data usage from workflows EGI Community Forum 2014, Helsinki, Finland 11

12 Data Avenue Blacktop Core service accessible through SOAP File transfers Directory operations Easy to add new protocols using the Adaptor interface HTTP(S), SFTP, SRM, GSIFTP, S3 EGI Community Forum 2014, Helsinki, Finland 12

13 Data Avenue Blacktop API Java API available: https://data-avenue.eu/webservice-api-doc Focused on easy usage when created EGI Community Forum 2014, Helsinki, Finland 13

14 Data Avenue Blacktop usage API or portlet ticket must be requested: https://data-avenue.eu/ticket-request-form Used to identify DA Blacktop clients EGI Community Forum 2014, Helsinki, Finland 14

15 Data Avenue Portlet Try it for yourself: https://data-avenue.eu/ Also available as a JSR-268 portlet (can be deployed over e.g. Liferay) Included in WS-PGRADE releases Two-panel layout Data up- and download Copy/move Favorites Progress monitoring 7/7/2014 Footer 15

16 Data SZTAKI 16

17 Data Avenue Liferay portlet 17

18 Data Avenue Liferay portlet 18

19 Data Avenue Liferay portlet 19

20 Data Avenue Liferay portlet 20

21 Generic data transfer among WS-PGRADE workflow nodes DCI1 WS-PGRADE Workflow DCI2 FS1 J2 J1 J4 J5 J3 FS2 FS3 FS5 The Data Avenue Blacktop services are available not only by the Data Avenue portlet but also by the nodes of a WS-PGRADE workflow J: Job FS: File storage system, e.g. gsiftp, irods, SRM

22 Data Avenue in WS- PGRADE/gUSE Data sources and destinations of jobs can be selected guse automatically manages data transfers using Blacktop Actual transfer delegated up to the worker node wherever possible (two-phase up- and download), bypassing the Blacktop service if the middleware is capable of handling the protocol To be released before summer EGI Community Forum 2014, Helsinki, Finland 22

23 Comparison with Globus Online 1. Globus Online is excellent inside a globus grid 2. But it supports only the Globus storage protocols 3. Does not enable to use inside a workflow 4. Data Avenue is a generalization of Globus Online 5. Enables the access to many different types of storages even in a workflow that runs through several kind of DCIs 6. This technology enables the easy integration of Globus and Cloud resources at workflow 23 level

24 Flexibility for collaboration among community members SHIWA Repository WF upload WF download guse Portal guse WF Repo guse Portal guse WF Repo Cloud 1 OpenNebula Cloud 2 Amazon Cloud n OpenStack 24

25 Flexibility in using different workflow systems Cyberspace WS-PGRADE Gateways Bio1 Bio2 BioN er Taverna Galaxy Kepler WF systems EMI Grids Glob us Infrastructures Cloud Combining SCI-BUS and SHIWA technologies (supported by ER- Flow) users can access and use many WFs and many infrastructures in an interoperable way no matter which is their home WF system 25

26 Flexibility of gateway types and user views 1. Generic purpose gateways for clouds (workflow view) Core WS-PGRADE/gUSE (e.g. Greek NGI) 2. Generic purpose gateway for specific technologies (workflow view) SHIWA gateway for workflow sharing and interoperation 3. Domain-specific science gateway instance Autodock gateway (end-user view) Swiss proteomics portal (customized GUI using ASM API) VisIVO Mobile (use of Remote API) 26

27 Some examples of SCI-BUS domain-specific gateways 27

28 The DRIHM project s gateway Other data sources

29 guse based gateways More than 100 deployments worldwide More than downloads from 75 countries on sourceforge 29

30 Conclusions Join SCI-BUS as associated member Why to select WS-PGRADE/gUSE and join the SCI-BUS community? 1.Robustness Already large number of gateways used in production 2.Sustainability The SCI-BUS project and its sustainability and commercialization plan guarantees it 3.Functionalities Rich functionalities that are growing according to the SCI-BUS and sourceforge community needs 4.How easy to adapt for the needs of the new user community? Already large number of gateways customized from guse/ws- PGRADE 5.You can influence the progress of WS-PGRADE/gUSE 30

31 Where to find further information? SCI-BUS web page: guse/ws-pgrade: guse on sourceforge 31

SCI-BUS gateways for grid and cloud infrastructures

SCI-BUS gateways for grid and cloud infrastructures SCI-BUS gateways for grid and cloud infrastructures Tamas Kiss University of Westminster Peter Kacsuk, Zoltan Farkas MTA SZTAKI VERCE project meeting 1 st February 2013, Edinburgh SCI-BUS is supported

More information

Data Avenue: Remote Storage Resource Management in WS-PGRADE/gUSE

Data Avenue: Remote Storage Resource Management in WS-PGRADE/gUSE Data Avenue: Remote Storage Resource Management in WS-PGRADE/gUSE Laboratory of Parallel and Distributed Systems Institute for Computer Science and Control Hungarian Academy of Sciences Budapest, Hungary

More information

Approaches for building an e-infrastructure: supercomputing and grid/cloud/crowd computing

Approaches for building an e-infrastructure: supercomputing and grid/cloud/crowd computing Approaches for building an e-infrastructure: supercomputing and grid/cloud/crowd computing Dr. Robert Lovas Deputy head of Laboratory of Parallel and Distributed Systems Coordinator of EU FP7 e-infrastructure

More information

Project Full Title: Cloud based Simulation platform for Manufacturing and Engineering. Project Acronym: CloudSME Project Number: 608886

Project Full Title: Cloud based Simulation platform for Manufacturing and Engineering. Project Acronym: CloudSME Project Number: 608886 Project Full Title: Cloud based Simulation platform for Manufacturing and Engineering Project Acronym: CloudSME Project Number: 608886 Programme: Cooperation Themes: Information and Communication Technologies;

More information

Hadoop Cloud SaaS access via WS-PGRADE adaptation

Hadoop Cloud SaaS access via WS-PGRADE adaptation Hadoop Cloud SaaS access via WS-PGRADE adaptation Elisa Cauhe 1 Arturo Giner 1 Jaime Ibar 1 Gonzalo Ruiz 1 Ruben Valles 1 BIFI: Institute for Biocomputation and Physics of Complex Systems of the University

More information

SHIWA is supported by the FP7 Capacities Programme under contract No. RI

SHIWA is supported by the FP7 Capacities Programme under contract No. RI SHaring Interoperable s for Large- Scale Scientific Simulations on Available s Interoperability Enables Interdisciplinary Research Gabor Terstyanszky, University of Westminster 02 nd February 2012 SHIWA

More information

User Manual: Using Hadoop with WS-PGRADE. workflow.

User Manual: Using Hadoop with WS-PGRADE. workflow. User Manual: Using Hadoop with WS-PGRADE workflows December 9, 2014 1 About This manual explains the configuration of a set of workflows that can be used to submit a Hadoop job through a WS-PGRADE portal.

More information

Anwendungsintegration und Workflows mit UNICORE 6

Anwendungsintegration und Workflows mit UNICORE 6 Mitglied der Helmholtz-Gemeinschaft Anwendungsintegration und Workflows mit UNICORE 6 Bernd Schuller und UNICORE-Team Jülich Supercomputing Centre, Forschungszentrum Jülich GmbH 26. November 2009 D-Grid

More information

Grid Engine. The EPIKH Project (Exchange Programme to advance e-infrastructure Know-How)

Grid Engine. The EPIKH Project (Exchange Programme to advance e-infrastructure Know-How) The EPIKH Project (Exchange Programme to advance e-infrastructure Know-How) Grid Engine Riccardo Rotondo (riccardo.rotondo@garr.it) Consortium GARR Joint CHAIN/EPIKH School for Application Porting to Science

More information

Open-source and Standards - Unleashing the Potential for Innovation of Cloud Computing

Open-source and Standards - Unleashing the Potential for Innovation of Cloud Computing Cloudscape IV Advances on Interoperability & Cloud Computing Standards Brussels, Belgium, February 23th, 2012 Open-source and Standards - Unleashing the Potential for Innovation of Cloud Computing Ignacio

More information

Test of cloud federation in CHAIN-REDS project

Test of cloud federation in CHAIN-REDS project Test of cloud federation in CHAIN-REDS project Italian National Institute of Nuclear Physics, Division of Catania - Italy E-mail: giuseppe.andronico@ct.infn.it Roberto Barbera Department of Physics and

More information

Zentrum für Informationsdienste und Hochleistungsrechnen (ZIH) A Data Driven Science Gateway for Computational Workflows

Zentrum für Informationsdienste und Hochleistungsrechnen (ZIH) A Data Driven Science Gateway for Computational Workflows Zentrum für Informationsdienste und Hochleistungsrechnen (ZIH) A Data Driven Science Gateway for Computational Workflows (richard.grunzke@tu-dresden.de) Introduction MoSGrid Science Gateway - Simple and

More information

PROGRESS Portal Access Whitepaper

PROGRESS Portal Access Whitepaper PROGRESS Portal Access Whitepaper Maciej Bogdanski, Michał Kosiedowski, Cezary Mazurek, Marzena Rabiega, Malgorzata Wolniewicz Poznan Supercomputing and Networking Center April 15, 2004 1 Introduction

More information

HPC Cloud Computing with OpenNebula

HPC Cloud Computing with OpenNebula High Performance Cloud Computing Day BiG Grid - SARA Amsterdam, The Netherland, October 4th, 2011 HPC Cloud Computing with OpenNebula Ignacio M. Llorente Project Director Acknowledgments The research leading

More information

Future Generation Computer Systems. Using a private desktop grid system for accelerating drug discovery

Future Generation Computer Systems. Using a private desktop grid system for accelerating drug discovery Future Generation Computer Systems 27 (2011) 657 666 Contents lists available at ScienceDirect Future Generation Computer Systems journal homepage: www.elsevier.com/locate/fgcs Using a private desktop

More information

Managing Complexity in Distributed Data Life Cycles Enhancing Scientific Discovery

Managing Complexity in Distributed Data Life Cycles Enhancing Scientific Discovery Center for Information Services and High Performance Computing (ZIH) Managing Complexity in Distributed Data Life Cycles Enhancing Scientific Discovery Richard Grunzke*, Jens Krüger, Sandra Gesing, Sonja

More information

A Survey Study on Monitoring Service for Grid

A Survey Study on Monitoring Service for Grid A Survey Study on Monitoring Service for Grid Erkang You erkyou@indiana.edu ABSTRACT Grid is a distributed system that integrates heterogeneous systems into a single transparent computer, aiming to provide

More information

Using the Grid for the interactive workflow management in biomedicine. Andrea Schenone BIOLAB DIST University of Genova

Using the Grid for the interactive workflow management in biomedicine. Andrea Schenone BIOLAB DIST University of Genova Using the Grid for the interactive workflow management in biomedicine Andrea Schenone BIOLAB DIST University of Genova overview background requirements solution case study results background A multilevel

More information

glibrary: Digital Asset Management System for the Grid

glibrary: Digital Asset Management System for the Grid glibrary: Digital Asset Management System for the Grid Antonio Calanducci INFN Catania EGEE User Forum Manchester, 09 th -11 th May 2007 www.eu-egee.org EGEE and glite are registered trademarks Outline

More information

Scientific and Technical Applications as a Service in the Cloud

Scientific and Technical Applications as a Service in the Cloud Scientific and Technical Applications as a Service in the Cloud University of Bern, 28.11.2011 adapted version Wibke Sudholt CloudBroker GmbH Technoparkstrasse 1, CH-8005 Zurich, Switzerland Phone: +41

More information

WS-PGRADE Portal Introduction

WS-PGRADE Portal Introduction WS-PGRADE Portal Introduction WS-PGRADE Customization: Portlets Eduardo Lostal Zaragoza, 17 th May 2012 Structure of the presentation SCI-BUS Development Liferay Portal Portlets Conclusion 2 What is being

More information

Orbiter Series Service Oriented Architecture Applications

Orbiter Series Service Oriented Architecture Applications Workshop on Science Agency Uses of Clouds and Grids Orbiter Series Service Oriented Architecture Applications Orbiter Project Overview Mark L. Green mlgreen@txcorp.com Tech-X Corporation, Buffalo Office

More information

The EGI pan-european Federation of Clouds

The EGI pan-european Federation of Clouds The EGI pan-european Federation of Clouds CGW12 Cracow, 22-24 Oct 2012 Matteo Turilli Senior Research Associate Chair EGI Federated Clouds Task Force Oxford e-research Centre University of Oxford matteo.turilli@oerc.ox.ac.uk

More information

Cluster, Grid, Cloud Concepts

Cluster, Grid, Cloud Concepts Cluster, Grid, Cloud Concepts Kalaiselvan.K Contents Section 1: Cluster Section 2: Grid Section 3: Cloud Cluster An Overview Need for a Cluster Cluster categorizations A computer cluster is a group of

More information

Widening the number of e- Infrastructure users with Science Gateways and Identity Federations (access for success)

Widening the number of e- Infrastructure users with Science Gateways and Identity Federations (access for success) Consorzio COMETA UNIONE EUROPEA Widening the number of e- Infrastructure users with Science Gateways and Identity Federations (access for success) Prof. Roberto Barbera (roberto.barbera@ct.infn.it) Univ.

More information

The evolution of Grid Brokers: union for interoperability

The evolution of Grid Brokers: union for interoperability The evolution of Grid Brokers: union for interoperability ATTILA KERTÉSZ Institute of Informatics, University of Szeged MTA SZTAKI Computer and Automation Research Institute keratt@inf.u-szeged.hu Reviewed

More information

Approaches for Cloud and Mobile Computing

Approaches for Cloud and Mobile Computing Joint CLEEN and ACROSS Workshop on Cloud Technology and Energy Efficiency in Mobile Communications at EUCNC 15, Paris, France - 29 June, 2015 Interoperable Data Management Approaches for Cloud and Mobile

More information

INDIGO DataCloud. Technical Overview RIA-653549. Giacinto.Donvito@ba.infn.it. INFN-Bari

INDIGO DataCloud. Technical Overview RIA-653549. Giacinto.Donvito@ba.infn.it. INFN-Bari INDIGO DataCloud Technical Overview RIA-653549 Giacinto.Donvito@ba.infn.it INFN-Bari Agenda Gap analysis Goals Architecture WPs activities Conclusions 2 Gap Analysis Support federated identities and provide

More information

Denis Caromel, CEO Ac.veEon. Orchestrate and Accelerate Applica.ons. Open Source Cloud Solu.ons Hybrid Cloud: Private with Burst Capacity

Denis Caromel, CEO Ac.veEon. Orchestrate and Accelerate Applica.ons. Open Source Cloud Solu.ons Hybrid Cloud: Private with Burst Capacity Cloud computing et Virtualisation : applications au domaine de la Finance Denis Caromel, CEO Ac.veEon Orchestrate and Accelerate Applica.ons Open Source Cloud Solu.ons Hybrid Cloud: Private with Burst

More information

E-Infrastructure Development Trends in the Area of Grids, Clouds, HPC, Storage, Virtualization and IaaS

E-Infrastructure Development Trends in the Area of Grids, Clouds, HPC, Storage, Virtualization and IaaS E-Infrastructure Development Trends in the Area of Grids, Clouds, HPC, Storage, Virtualization and IaaS Peter Kacsuk, MTA-SZTAKI, kacsuk@sztaki.hu Peter Stefan, NIIFI, stefan@niif.hu Imre Szeberenyi, BME,

More information

The EDGeS project receives Community research funding

The EDGeS project receives Community research funding Desktop Grids EDGeS project Delegation for access to trusted resources The EDGeS project receives Community research funding 1 DG = Desktop Grid = Loose grid scavenging idle resources Unit of Work = Application

More information

Cloud security monitoring and vulnerability management

Cloud security monitoring and vulnerability management Cloud security monitoring and vulnerability management M. Kozlovszky*, L.Kovács*, M. Törőcsik*, G.Windisch*, S.Ács*, D.Prém*, Gy. Eigner*, P.I. Sas*, T. Schubert*, V. Póserné* * Óbuda University/John von

More information

Grids Computing and Collaboration

Grids Computing and Collaboration Grids Computing and Collaboration Arto Teräs CSC, the Finnish IT center for science University of Pune, India, March 12 th 2007 Grids Computing and Collaboration / Arto Teräs 2007-03-12 Slide

More information

An approach to grid scheduling by using Condor-G Matchmaking mechanism

An approach to grid scheduling by using Condor-G Matchmaking mechanism An approach to grid scheduling by using Condor-G Matchmaking mechanism E. Imamagic, B. Radic, D. Dobrenic University Computing Centre, University of Zagreb, Croatia {emir.imamagic, branimir.radic, dobrisa.dobrenic}@srce.hr

More information

GridWay: Open Source Meta-scheduling Technology for Grid Computing

GridWay: Open Source Meta-scheduling Technology for Grid Computing : Open Source Meta-scheduling Technology for Grid Computing Ruben S. Montero dsa-research.org Open Source Grid & Cluster Oakland CA, May 2008 Contents Introduction What is? Architecture & Components Scheduling

More information

Execution of scientific workflows on federated multi-cloud infrastructures

Execution of scientific workflows on federated multi-cloud infrastructures Execution of scientific workflows on federated multi-cloud infrastructures Daniele Lezzi 1, Francesc Lordan 1, Roger Rafanell 1, and Rosa M. Badia 1, 1 Barcelona Supercomputing Center - Centro Nacional

More information

New resource provision paradigms for Grid Infrastructures: Virtualization and Cloud

New resource provision paradigms for Grid Infrastructures: Virtualization and Cloud CISCO NerdLunch Series November 7, 2008 San Jose, CA New resource provision paradigms for Grid Infrastructures: Virtualization and Cloud Ruben Santiago Montero Distributed Systems Architecture Research

More information

Hybrid Distributed Computing Infrastructure Experiments in Grid5000: Supporting QoS in Desktop Grids with Cloud Resources

Hybrid Distributed Computing Infrastructure Experiments in Grid5000: Supporting QoS in Desktop Grids with Cloud Resources Hybrid Distributed Computing Infrastructure Experiments in Grid5: Supporting QoS in Desktop Grids with Cloud Resources Simon Delamare, Gilles Fedak and Oleg Lodygensky simon.delamare@inria.fr - gilles.fedak@inria.fr

More information

Software Entwicklungen für das LSDF Datenmanagement

Software Entwicklungen für das LSDF Datenmanagement Software Entwicklungen für das LSDF Datenmanagement Rainer Stotzka, V. Hartmann, T. Jejkal,, P. Neuberger, S. Ochsenreither, F. Rindone, T. Schmidt, H. Pasic J. van Wezel, A. Garcia, R. Kupsch, S. Bourov,

More information

Grid Scheduling Architectures with Globus GridWay and Sun Grid Engine

Grid Scheduling Architectures with Globus GridWay and Sun Grid Engine Grid Scheduling Architectures with and Sun Grid Engine Sun Grid Engine Workshop 2007 Regensburg, Germany September 11, 2007 Ignacio Martin Llorente Javier Fontán Muiños Distributed Systems Architecture

More information

Volunteer Computing, Grid Computing and Cloud Computing: Opportunities for Synergy. Derrick Kondo INRIA, France

Volunteer Computing, Grid Computing and Cloud Computing: Opportunities for Synergy. Derrick Kondo INRIA, France Volunteer Computing, Grid Computing and Cloud Computing: Opportunities for Synergy Derrick Kondo INRIA, France Outline Cloud Grid Volunteer Computing Cloud Background Vision Hide complexity of hardware

More information

The Lattice Project: A Multi-Model Grid Computing System. Center for Bioinformatics and Computational Biology University of Maryland

The Lattice Project: A Multi-Model Grid Computing System. Center for Bioinformatics and Computational Biology University of Maryland The Lattice Project: A Multi-Model Grid Computing System Center for Bioinformatics and Computational Biology University of Maryland Parallel Computing PARALLEL COMPUTING a form of computation in which

More information

for my computation? Stefano Cozzini Which infrastructure Which infrastructure Democrito and SISSA/eLAB - Trieste

for my computation? Stefano Cozzini Which infrastructure Which infrastructure Democrito and SISSA/eLAB - Trieste Which infrastructure Which infrastructure for my computation? Stefano Cozzini Democrito and SISSA/eLAB - Trieste Agenda Introduction:! E-infrastructure and computing infrastructures! What is available

More information

XSEDE Service Provider Software and Services Baseline. September 24, 2015 Version 1.2

XSEDE Service Provider Software and Services Baseline. September 24, 2015 Version 1.2 XSEDE Service Provider Software and Services Baseline September 24, 2015 Version 1.2 i TABLE OF CONTENTS XSEDE Production Baseline: Service Provider Software and Services... i A. Document History... A-

More information

Orchestrated service deployment, maintenance, and debugging in IaaS clouds for crowd computing *

Orchestrated service deployment, maintenance, and debugging in IaaS clouds for crowd computing * Orchestrated service deployment, maintenance, and debugging in IaaS clouds for crowd computing * Robert Lovas Institute for Computer Science and Control, Hungarian Academy of Sciences Kende u. 13-17, Budapest

More information

Concepts and Architecture of the Grid. Summary of Grid 2, Chapter 4

Concepts and Architecture of the Grid. Summary of Grid 2, Chapter 4 Concepts and Architecture of the Grid Summary of Grid 2, Chapter 4 Concepts of Grid Mantra: Coordinated resource sharing and problem solving in dynamic, multi-institutional virtual organizations Allows

More information

Deploying Business Virtual Appliances on Open Source Cloud Computing

Deploying Business Virtual Appliances on Open Source Cloud Computing International Journal of Computer Science and Telecommunications [Volume 3, Issue 4, April 2012] 26 ISSN 2047-3338 Deploying Business Virtual Appliances on Open Source Cloud Computing Tran Van Lang 1 and

More information

UFTP High-performance data transfer for UNICORE

UFTP High-performance data transfer for UNICORE Mitglied der Helmholtz-Gemeinschaft UFTP High-performance data transfer for UNICORE Dr. Bernd Schuller, Tim Pohlmann Federated Systems and Data division Jülich Supercomputer Centre Forschungszentrum Jülich

More information

Vangelis Floros, GRNET S.A. 3 rd Open Source Software Conference March 22, 2008 NTUA, Athens Greece

Vangelis Floros, GRNET S.A. 3 rd Open Source Software Conference March 22, 2008 NTUA, Athens Greece Vangelis Floros, GRNET S.A. 3 rd Open Source Software Conference March 22, 2008 NTUA, Athens Greece Introduction What is a Grid? What is escience? Large Scientific Grids The example of EGEE Building Grid

More information

Enhancing UNICORE Storage Management using Hadoop

Enhancing UNICORE Storage Management using Hadoop Enhancing UNICORE Storage Management using Hadoop Distributed ib t File System Wasim Bari 2, Ahmed Shiraz Memon 1, Dr. Bernd Schuller 1 1. Jülich Supercomputing Centre, Forschungszentrum Jülich & 2. Institute

More information

User Guide of edox Archiver, the Electronic Document Handling Gateway of

User Guide of edox Archiver, the Electronic Document Handling Gateway of User Guide of edox Archiver, the Electronic Document Handling Gateway of project v0.7 SCI-BUS is supported by the FP7 Capacities Programme under contract nr RI-283481 Table of Contents 1 INTRODUCTION...

More information

MIGRATING DESKTOP AND ROAMING ACCESS. Migrating Desktop and Roaming Access Whitepaper

MIGRATING DESKTOP AND ROAMING ACCESS. Migrating Desktop and Roaming Access Whitepaper Migrating Desktop and Roaming Access Whitepaper Poznan Supercomputing and Networking Center Noskowskiego 12/14 61-704 Poznan, POLAND 2004, April white-paper-md-ras.doc 1/11 1 Product overview In this whitepaper

More information

HISP: a data-driven portal for hadron therapy

HISP: a data-driven portal for hadron therapy HISP: a data-driven portal for hadron therapy Faustin Laurentiu Roman CERN / IFIC Prototype architecture Tools, implementation & services Conclusions (& demo) 1 One slide situation: ereferral and escience

More information

Australian Synchrotron, Storage Gateway

Australian Synchrotron, Storage Gateway Australian Synchrotron, Storage Gateway User Help Manual Version 1.2 Storage Gateway User Help Manual 2 REVISION HISTORY Date Version Description Author 2 May 2008 1.0 Document creation Chris Myers 13

More information

EMI Storage meets EMI security

EMI Storage meets EMI security EMI Storage meets EMI security Component/ Middleware glite (LFC,FTS,DPM,GFAL) ARC UNICORE StoRM dcache Staff With kind contributions by Oliver Keeble, Jean- Philippe Baud Jon Kerr Nilsen Ralph Müller-

More information

EXTENDING THE SHEBA PROPAGATIO MODEL TO REDUCE PARAMETER-RELATED UNCERTAINTIES

EXTENDING THE SHEBA PROPAGATIO MODEL TO REDUCE PARAMETER-RELATED UNCERTAINTIES Computer Science 14 (2) 2013 http://dx.doi.org/10.7494/csci.2013.14.2.253 Gabriele Pierantoni Brian Coghlan Eamonn Kenny Peter Gallagher David Pérez-Suárez EXTENDING THE SHEBA PROPAGATIO MODEL TO REDUCE

More information

The GISELA Science Gateway

The GISELA Science Gateway The GISELA Science Gateway Roberto Barbera (roberto.barbera@ct.infn.it) University of Catania and INFN - Italy TICAL 2012 Lima, 3 July 2012 Introduction and driving considerations The Catania Science Gateway

More information

Cloud-pilot.doc 12-12-2010 SA1 Marcus Hardt, Marcin Plociennik, Ahmad Hammad, Bartek Palak E U F O R I A

Cloud-pilot.doc 12-12-2010 SA1 Marcus Hardt, Marcin Plociennik, Ahmad Hammad, Bartek Palak E U F O R I A Identifier: Date: Activity: Authors: Status: Link: Cloud-pilot.doc 12-12-2010 SA1 Marcus Hardt, Marcin Plociennik, Ahmad Hammad, Bartek Palak E U F O R I A J O I N T A C T I O N ( S A 1, J R A 3 ) F I

More information

Welcome. Grid on Demand. Willem Toorop and Alain van Hoof. {wtoorop,ahoof}@os3.nl. June 30, 2010

Welcome. Grid on Demand. Willem Toorop and Alain van Hoof. {wtoorop,ahoof}@os3.nl. June 30, 2010 Welcome Grid on Demand Willem Toorop and Alain van Hoof {wtoorop,ahoof}@os3.nl June 30, 2010 Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 1 / 39 Research Question Introduction Research

More information

EUFORIA: Grid and High Performance Computing at the Service of Fusion Modelling

EUFORIA: Grid and High Performance Computing at the Service of Fusion Modelling EUFORIA: Grid and High Performance Computing at the Service of Fusion Modelling Miguel Cárdenas-Montes on behalf of Euforia collaboration Ibergrid 2008 May 12 th 2008 Porto Outline Project Objectives Members

More information

Project Full Title: Cloud based Simulation platform for Manufacturing and Engineering. Project Acronym: CloudSME Project Number: 608886

Project Full Title: Cloud based Simulation platform for Manufacturing and Engineering. Project Acronym: CloudSME Project Number: 608886 Project Full Title: Cloud based Simulation platform for Manufacturing and Engineering Project Acronym: CloudSME Project Number: 608886 Programme: Cooperation Themes: Information and Communication Technologies;

More information

THE CCLRC DATA PORTAL

THE CCLRC DATA PORTAL THE CCLRC DATA PORTAL Glen Drinkwater, Shoaib Sufi CCLRC Daresbury Laboratory, Daresbury, Warrington, Cheshire, WA4 4AD, UK. E-mail: g.j.drinkwater@dl.ac.uk, s.a.sufi@dl.ac.uk Abstract: The project aims

More information

Deploying Multiscale Applications on European e-infrastructures

Deploying Multiscale Applications on European e-infrastructures Deploying Multiscale Applications on European e-infrastructures 04/06/2013 Ilya Saverchenko The MAPPER project receives funding from the EC's Seventh Framework Programme (FP7/2007-2013) under grant agreement

More information

Data Storage Solution Using PL-Grid UNICORE Infrastructure

Data Storage Solution Using PL-Grid UNICORE Infrastructure Domain-oriented services and resources of Polish Infrastructure for Supporting Computational Science in the European Research Space PLGrid Plus Data Storage Solution Using PL-Grid UNICORE Infrastructure

More information

Improvements of the grid infrastructure and services within SEE-GRID

Improvements of the grid infrastructure and services within SEE-GRID Improvements of the grid infrastructure and services within SEE-GRID Anastas Misev MARNET/MARGI/UKIM Macedonia Introduction SEE-GRID Project series SEE-GRID establish infrastructure SEE-GRID-2 extend infrastructure,

More information

Living in a mixed world -Interoperability in Windows HPC Server 2008. Steven Newhouse stevenn@microsoft.com

Living in a mixed world -Interoperability in Windows HPC Server 2008. Steven Newhouse stevenn@microsoft.com Living in a mixed world -Interoperability in Windows HPC Server 2008 Steven Newhouse stevenn@microsoft.com Overview Scenarios: Mixed Environments Authentication & Authorization File Systems Application

More information

SOA REFERENCE ARCHITECTURE: WEB TIER

SOA REFERENCE ARCHITECTURE: WEB TIER SOA REFERENCE ARCHITECTURE: WEB TIER SOA Blueprint A structured blog by Yogish Pai Web Application Tier The primary requirement for this tier is that all the business systems and solutions be accessible

More information

Cloud Computing Architecture with OpenNebula HPC Cloud Use Cases

Cloud Computing Architecture with OpenNebula HPC Cloud Use Cases NASA Ames NASA Advanced Supercomputing (NAS) Division California, May 24th, 2012 Cloud Computing Architecture with OpenNebula HPC Cloud Use Cases Ignacio M. Llorente Project Director OpenNebula Project.

More information

GRID COMPUTING Techniques and Applications BARRY WILKINSON

GRID COMPUTING Techniques and Applications BARRY WILKINSON GRID COMPUTING Techniques and Applications BARRY WILKINSON Contents Preface About the Author CHAPTER 1 INTRODUCTION TO GRID COMPUTING 1 1.1 Grid Computing Concept 1 1.2 History of Distributed Computing

More information

Project Title: Judicial Branch Enterprise Document Management System RFP Number: FIN122210CK DMS TECHNICAL REQUIREMENTS

Project Title: Judicial Branch Enterprise Document Management System RFP Number: FIN122210CK DMS TECHNICAL REQUIREMENTS APPENDIX C. DMS TECHNICAL REQUIREMENTS C.1 INTRODUCTION This appendix is an overview of the California Judicial Branch infrastructure and technical requirements considerations that require a written response.

More information

SEERA-EI. Introduction to Cloud Computing. www.seera-ei.eu. SEERA-EI training, 13 April 2011. Aneta Karaivanova, IICT-BAS, Bulgaria

SEERA-EI. Introduction to Cloud Computing. www.seera-ei.eu. SEERA-EI training, 13 April 2011. Aneta Karaivanova, IICT-BAS, Bulgaria SEERA-EI Introduction to Cloud Computing www.seera-ei.eu SEERA-EI training, 13 April 2011 Aneta Karaivanova, IICT-BAS, Bulgaria The pan-eu e-infrastructures vision The Research Network infrastructure provides

More information

IGI Portal architecture and interaction with a CA- online

IGI Portal architecture and interaction with a CA- online IGI Portal architecture and interaction with a CA- online Abstract In the framework of the Italian Grid Infrastructure, we are designing a web portal for the grid and cloud services provisioning. In following

More information

A Service for Data-Intensive Computations on Virtual Clusters

A Service for Data-Intensive Computations on Virtual Clusters A Service for Data-Intensive Computations on Virtual Clusters Executing Preservation Strategies at Scale Rainer Schmidt, Christian Sadilek, and Ross King rainer.schmidt@arcs.ac.at Planets Project Permanent

More information

International Symposium on Grid Computing 2009 April 23th, Academia Sinica, Taipei, Taiwan

International Symposium on Grid Computing 2009 April 23th, Academia Sinica, Taipei, Taiwan International Symposium on Grid Computing 2009 April 23th, Academia Sinica, Taipei, Taiwan New resource provision paradigms for Grid Infrastructures: Virtualization and Cloud Ruben Santiago Montero Distributed

More information

Concepts and Architecture of Grid Computing. Advanced Topics Spring 2008 Prof. Robert van Engelen

Concepts and Architecture of Grid Computing. Advanced Topics Spring 2008 Prof. Robert van Engelen Concepts and Architecture of Grid Computing Advanced Topics Spring 2008 Prof. Robert van Engelen Overview Grid users: who are they? Concept of the Grid Challenges for the Grid Evolution of Grid systems

More information

Building Platform as a Service for Scientific Applications

Building Platform as a Service for Scientific Applications Building Platform as a Service for Scientific Applications Moustafa AbdelBaky moustafa@cac.rutgers.edu Rutgers Discovery Informa=cs Ins=tute (RDI 2 ) The NSF Cloud and Autonomic Compu=ng Center Department

More information

File transfer in UNICORE State of the art

File transfer in UNICORE State of the art Mitglied der Helmholtz-Gemeinschaft File transfer in UNICORE State of the art Bernd Schuller, Björn Hagemeier, Michael Rambadt Federated Systems and Data division Jülich Supercomputer Centre Forschungszentrum

More information

The glite File Transfer Service

The glite File Transfer Service Enabling Grids Enabling for E-sciencE Grids for E-sciencE The glite File Transfer Service Paolo Badino On behalf of the JRA1 Data Management team EGEE User Forum - CERN, 2 Mars 2006 www.eu-egee.org Outline

More information

Open Source Cloud Computing Management with OpenNebula

Open Source Cloud Computing Management with OpenNebula CloudCamp Campus Party July 2011, Valencia Open Source Cloud Computing Management with OpenNebula Javier Fontán Muiños dsa-research.org Distributed Systems Architecture Research Group Universidad Complutense

More information

MassTransit vs. FTP Comparison

MassTransit vs. FTP Comparison MassTransit vs. Comparison If you think is an optimal solution for delivering digital files and assets important to the strategic business process, think again. is designed to be a simple utility for remote

More information

Recap& What%is% big%data % Introduction%to%e4science% Reflections%of% big%data %in%health% Guest%lecture:%% DB%for%big%data%analytics%

Recap& What%is% big%data % Introduction%to%e4science% Reflections%of% big%data %in%health% Guest%lecture:%% DB%for%big%data%analytics% Recap& What%is% big%data % Introduction%to%e4science% Reflections%of% big%data %in%health% Guest%lecture:%% DB%for%big%data%analytics% 1$ Recap& Data%infrastructure% Sources,%capture,%representation,%access%

More information

Communiqué 4. Standardized Global Content Management. Designed for World s Leading Enterprises. Industry Leading Products & Platform

Communiqué 4. Standardized Global Content Management. Designed for World s Leading Enterprises. Industry Leading Products & Platform Communiqué 4 Standardized Communiqué 4 - fully implementing the JCR (JSR 170) Content Repository Standard, managing digital business information, applications and processes through the web. Communiqué

More information

TUTORIAL. Rebecca Breu, Bastian Demuth, André Giesler, Bastian Tweddell (FZ Jülich) {r.breu, b.demuth, a.giesler, b.tweddell}@fz-juelich.

TUTORIAL. Rebecca Breu, Bastian Demuth, André Giesler, Bastian Tweddell (FZ Jülich) {r.breu, b.demuth, a.giesler, b.tweddell}@fz-juelich. TUTORIAL Rebecca Breu, Bastian Demuth, André Giesler, Bastian Tweddell (FZ Jülich) {r.breu, b.demuth, a.giesler, b.tweddell}@fz-juelich.de September 2006 Outline Motivation & History Production UNICORE

More information

On Enabling Hydrodynamics Data Analysis of Analytical Ultracentrifugation Experiments

On Enabling Hydrodynamics Data Analysis of Analytical Ultracentrifugation Experiments On Enabling Hydrodynamics Data Analysis of Analytical Ultracentrifugation Experiments 18. June 2013 Morris Reidel, Shahbaz Memon, et al. Outline Background Ultrascan Application Ultrascan Software Components

More information

GridFTP: A Data Transfer Protocol for the Grid

GridFTP: A Data Transfer Protocol for the Grid GridFTP: A Data Transfer Protocol for the Grid Grid Forum Data Working Group on GridFTP Bill Allcock, Lee Liming, Steven Tuecke ANL Ann Chervenak USC/ISI Introduction In Grid environments,

More information

Agenda: 1. Background 2. Solution: ProActive 3. Live Demonstration 4. IFP EN Use Case

Agenda: 1. Background 2. Solution: ProActive 3. Live Demonstration 4. IFP EN Use Case Advances in Cloud Computing with ProActive Parallel Suite D. Caromel Accelerate and Orchestrate Enterprise Applications Hybrid Cloud Solutions (Private with Public Burst) Agenda: 1. Background 2. Solution:

More information

The XSEDE Global Federated File System (GFFS) - Breaking Down Barriers to Secure Resource Sharing

The XSEDE Global Federated File System (GFFS) - Breaking Down Barriers to Secure Resource Sharing December 19, 2013 The XSEDE Global Federated File System (GFFS) - Breaking Down Barriers to Secure Resource Sharing Andrew Grimshaw, University of Virginia Co-architect XSEDE The complexity of software

More information

Plateforme de Calcul pour les Sciences du Vivant. SRB & glite. V. Breton. http://clrpcsv.in2p3.fr

Plateforme de Calcul pour les Sciences du Vivant. SRB & glite. V. Breton. http://clrpcsv.in2p3.fr SRB & glite V. Breton http://clrpcsv.in2p3.fr Introduction Goal: evaluation of existing technologies for data and tools integration and deployment Data and tools integration should be addressed using web

More information

Workspaces Concept and functional aspects

Workspaces Concept and functional aspects Mitglied der Helmholtz-Gemeinschaft Workspaces Concept and functional aspects A You-tube for science inspired by the High Level Expert Group Report on Scientific Data 21.09.2010 Morris Riedel, Peter Wittenburg,

More information

The PhysiomeSpace data sharing service and the new VPH-Share end-user interface

The PhysiomeSpace data sharing service and the new VPH-Share end-user interface The PhysiomeSpace data sharing service and the new end-user interface Matteo Balasso SCS srl, Italy Outline Introduction Past experience: PhysiomeSpace user interface Knowledge sharing Sharing data, models

More information

File Transfer Software and Service SC3

File Transfer Software and Service SC3 File Transfer Software and Service SC3 Gavin McCance JRA1 Data Management Cluster Service Challenge Meeting April 26 2005, Taipei www.eu-egee.org Outline Overview of Components Tier-0 / Tier-1 / Tier-2

More information

Future Generation Computer Systems

Future Generation Computer Systems Future Generation Computer Systems 29 (2013) 1442 1451 Contents lists available at SciVerse ScienceDirect Future Generation Computer Systems journal homepage: www.elsevier.com/locate/fgcs Towards a volunteer

More information

QuakeSim Project: Portals and Web Services for Geo-Sciences

QuakeSim Project: Portals and Web Services for Geo-Sciences CTS 2008 Tutorial II QuakeSim Project: Portals and Web Services for Geo-Sciences Marlon Pierce Indiana University mpierce@cs.indiana.edu QuakeSim Project Summary Goal is to provide a distributed environment

More information

Accessing the FTP Server - User Manual

Accessing the FTP Server - User Manual CENTRAL BANK OF CYPRUS Accessing the FTP Server - User Manual IT Department, CENTRAL BANK OF CYPRUS TABLE OF CONTENTS 1 EXECUTIVE SUMMARY... 1 1.1 AUDIENCE... 1 1.2 SCOPE... 1 2 CHANGES FROM THE OLD FTP

More information

HPC and Grid Concepts

HPC and Grid Concepts HPC and Grid Concepts Divya MG (divyam@cdac.in) CDAC Knowledge Park, Bangalore 16 th Feb 2012 GBC@PRL Ahmedabad 1 Presentation Overview What is HPC Need for HPC HPC Tools Grid Concepts GARUDA Overview

More information

HPC Cloud Computing Guide. www.penguincomputing.com 1-888-PENGUIN (736-4846) twitter: @Penguin HPC

HPC Cloud Computing Guide. www.penguincomputing.com 1-888-PENGUIN (736-4846) twitter: @Penguin HPC HPC Cloud Computing Guide www.penguincomputing.com 1888PENGUIN (7364846) twitter: @Penguin HPC organizations are facing increasing pressure to deliver critical services to their users while their budgets

More information

Data Management System for grid and portal services

Data Management System for grid and portal services Data Management System for grid and portal services Piotr Grzybowski 1, Cezary Mazurek 1, Paweł Spychała 1, Marcin Wolski 1 1 Poznan Supercomputing and Networking Center, ul. Noskowskiego 10, 61-704 Poznan,

More information

Program Grid and HPC5+ workshop

Program Grid and HPC5+ workshop Program Grid and HPC5+ workshop 24-30, Bahman 1391 Tuesday Wednesday 9.00-9.45 9.45-10.30 Break 11.00-11.45 11.45-12.30 Lunch 14.00-17.00 Workshop Rouhani Karimi MosalmanTabar Karimi G+MMT+K Opening IPM_Grid

More information

HPC Cluster Decisions and ANSYS Configuration Best Practices. Diana Collier Lead Systems Support Specialist Houston UGM May 2014

HPC Cluster Decisions and ANSYS Configuration Best Practices. Diana Collier Lead Systems Support Specialist Houston UGM May 2014 HPC Cluster Decisions and ANSYS Configuration Best Practices Diana Collier Lead Systems Support Specialist Houston UGM May 2014 1 Agenda Introduction Lead Systems Support Specialist Cluster Decisions Job

More information