GridFTP: A Data Transfer Protocol for the Grid
|
|
- Barbara Barber
- 8 years ago
- Views:
Transcription
1 GridFTP: A Data Transfer Protocol for the Grid Grid Forum Data Working Group on GridFTP Bill Allcock, Lee Liming, Steven Tuecke ANL Ann Chervenak USC/ISI <tuecke@mcs.anl.gov> Introduction In Grid environments, access to distributed data is typically as important as access to distributed computational resources. Distributed scientific and engineering applications require: transfers of large amounts of data (terabytes or petabytes) between storage systems, and access to large amounts of data (gigabytes or terabytes) by many geographically distributed applications and users for analysis, visualization, etc. Unfortunately, the lack of standard protocols for transfer and access of data in the Grid has led to a fragmented Grid storage community. Users who wish to access different storage systems are forced to use multiple protocols and/or APIs, and it is difficult to efficiently transfer data between these different storage systems. We propose a common data transfer and access protocol called GridFTP that provides secure, efficient data movement in Grid environments. This protocol, which extends the standard FTP protocol, provides a superset of the features offered by the various Grid storage systems currently in use. We chose the FTP protocol because it is the most commonly used protocol for data transfer on the Internet, and of the existing candidates from which to start it comes closest to meeting the Grid s needs. The GridFTP protocol includes the following features: Grid Security Infrastructure (GSI) and Kerberos support Third-party control of data transfer Parallel data transfer Striped data transfer Partial file transfer Automatic negotiation of TCP buffer/window sizes Support for reliable and restartable data transfer Integrated instrumentation Some of these features are supported by FTP extensions that have already been standardized in the IETF, but which are currently seldom implemented. Other features are new extensions FTP. 1
2 The Grid Forum GridFTP working group was formed within the Data working group to foster the development of a GridFTP protocol standard, as well as interoperable storage clients and servers that implement the GridFTP protocol. Motivation for a common protocol There are already a number of storage systems in use by the Grid community. These storage systems have been created in response to specific needs for storing and accessing large datasets. They each focus on a distinct set of requirements and provide distinct services to their clients. For example, some storage systems (DPSS, HPSS) focus on high-performance access to data and utilize parallel data transfer streams and/or striping across multiple servers to improve performance. i ii Other systems (DFS) focus on supporting high-volume usage and utilize dataset replication and local caching to divide and balance server load. iii The SRB system connects heterogeneous data collections and provides a uniform client interface to these repositories, and also provides metadata for use in identifying and locating data within the storage system. iv Still other systems (HDF5) focus on the structure of the data, and provide client support for accessing structured data from a variety of underlying storage systems. v Unfortunately, most of these storage systems utilize incompatible, an often unpublished protocols for accessing data, and therefore require the use of their own client libraries to access data. The use of multiple incompatible protocols and client libraries for accessing storage effectively partitions the datasets available on the grid. Applications that require access to data stored in different storage systems must either choose to only use a subset of storage systems, or must use multiple methods to retrieve data from the various storage systems. One approach to breaking down partitions created by these mutually incompatible storage system protocols is to build a layered client or gateway which can present the user with one interface, but which translates requests into the various storage system protocols and/or client library calls. This approach is attractive to existing storage system providers because it does not require them adopt support for a new protocol. But it also has significant disadvantages, including: Performance: Costly translations are often required between the layered client and storage system specific client libraries and protocols. In addition, it can be challenging to efficiently transfer a dataset from one storage system to another. Complexity: Building and maintaining a client or gateway that supports numerous storage systems is considerable work. In addition, staying up to date as each storage system independently evolves is very difficult. This is further exacerbated by the need to provide support for multiple client languages, such as C/C++, Java, Perl, Python, shells, etc. It would be mutually advantageous to both storage providers and users to have a common level of interoperability between all of these disparate systems: a common but extensible underlying data transfer protocol. Storage providers would gain a broader user base, because their data would be available to any client. Storage users would gain access to a broader range of storage systems and data. In addition, these benefits can be gained without the performance and complexity problems of the layered client or gateway approach. 2
3 Furthermore, establishing a common data transfer protocol would eliminate the current duplication of effort in developing unique data transfer capabilities for different storage systems. A pooling of effort in the data transfer protocol area would lead to greater reliability, performance, and overall features that would then be available to all distributed storage systems. Characteristics of the data transfer protocol In order to make a common data transfer protocol attractive to users and developers of existing storage systems, we must provide a transfer protocol that offers a superset of the features offered by systems currently in regular use. In addition, the protocol must be extensible, in order to support future innovations by storage system users and developers. We have observed that the FTP protocol is the protocol most commonly used for data transfer on the Internet, and the most likely candidate for meeting the Grid s needs. It is attractive in particular for the following reasons. It is a widely implemented and well-understood IETF standard protocol. There is a large code base and expertise from which to build. It provides a well-defined architecture for protocol extensions, and supports dynamic discovery of the extensions supported by a particular implementation. Numerous groups have added various extensions through the IETF. Some of these extensions are particularly useful in the Grid. In addition to client/server transfers (i.e. put/get, or remote read/write ), it also supports transfers directly between two servers, mediated by a third party client (i.e. third party transfer ). The separation of data and control channels onto different sockets allows for easier extensibility for parallel and striped transfers, efficiently transiting firewalls, etc. Most current FTP implementations support only a subset of the features defined in the FTP protocol (RFC 969) and its accepted extensions. Some of the seldom-implemented features are useful to Grid applications. But the standards also lack several features which Grid applications require. We intend to select a subset of the existing FTP standards and further extend it, adding the following features. We believe that the resulting protocol will be a suitable candidate for the common data transfer protocol for the grid, which we call GridFTP. Grid Security Infrastructure (GSI) and Kerberos support Robust and flexible authentication, integrity, and confidentiality features are critical when transferring or accessing files. GridFTP must support GSI and Kerberos authentication, with user controlled setting of various levels of data integrity and/or confidentiality. GridFTP provides this capability by implementing the GSSAPI authentication mechanisms defined by RFC 2228, FTP Security Extensions. Third-party control of data transfer In order to manage large data sets for large distributed communities, it is necessary to provide third-party control of transfers between storage servers. GridFTP provides this capability by 3
4 adding GSSAPI security to the existing third-party transfer capability defined in the FTP standard. Parallel data transfer On wide-area links, using multiple TCP streams (even between the same source and destination) can improve aggregate bandwidth over using a single TCP stream. This is required both between a single client and a single server, and between two servers. GridFTP supports parallel data transfer through FTP command extensions and data channel extensions defined in the Grid Forum draft. Striped data transfer Using multiple TCP streams to transfer data that is partitioned across multiple servers (ala. DPSS) can further improve aggregate bandwidth. GridFTP supports striped data transfers through extensions defined in the Grid Forum draft. Partial file transfer Many applications require the transfer of partial files. However, standard FTP requires the application to transfer the entire file, or the remainder of a file starting at a particular offset. GridFTP introduces new FTP commands, as defined in the Grid Forum draft, to support transfers of regions of a file. Automatic negotiation of TCP buffer/window sizes Manually setting TCP buffer/window sizes is an error-prone process (particularly for nonexperts) and is often simply not done. GridFTP extends the standard FTP command set and data channel protocol to support both manual setting and automatic negotiation of TCP buffer sizes both for large files and large sets of small files. Support for reliable data transfer Reliable transfer is important for many applications that manage data. Fault recovery methods for handling transient network failures, server outages, etc. are needed. The FTP standard includes basic features for restarting failed transfer that are not widely implemented. The GridFTP protocol exploits these features, and extends them cover the new data channel protocol. Implementation Status The GridFTP protocol has been adopted, in full or in part, in the following software systems: Globus Toolkit: The next version of the Globus Toolkit, which is currently available as an alpha release, includes a reference implementation of the full GridFTP protocol. Client and server SDKs, command line programs, and sample servers are included. wuftpd: The Washington University ftpd server has been modified to support GSI, partial files, and parallel transfers. This work was done by the ACES group at NCSA and the Globus Project group at Argonne. ncftp: The ncftp client has been modified to support GSI. This work was done by the Globus Project group at Argonne. 4
5 HPSS pftpd: The HPSS pftpd server has been modified to support GSI. This work was done by SDSC. Unitree ftpd: The Unitree ftpd server has been modified to support GSI. This work was done by the ACES group at NCSA. References i B. Tierney, W. Johnston, J. Lee, G. Hoo, and M. Thompson. End-to-end performance analysis of high speed distributed storage systems in wide area ATM networ ks. In NASA/Goddard Conference on Mass Storage Systems and Technologies, 1996, LBNL ii R.W. Watson and R.A. Coyne. The parallel I/O architecture of the high-performance storage system (HPSS). In IEEE MSS Symposium, iii It is interesting to note that DFS also provides for cache coherence in the face of multiple writers. This is one of its drawbacks for data grid applications, since this feature forces unnecessary overhead. Data grid applications seldom need the ability to change existing datasets. iv Information on SRB is available on the World Wide Web at v Information about HDF/HDF5 is available on the World Wide Web at vi Information about the Globus gsiftp is available on the World Wide Web at 5
Information Sciences Institute University of Southern California Los Angeles, CA 90292 {annc, carl}@isi.edu
_ Secure, Efficient Data Transport and Replica Management for High-Performance Data-Intensive Computing Bill Allcock 1 Joe Bester 1 John Bresnahan 1 Ann L. Chervenak 2 Ian Foster 1,3 Carl Kesselman 2 Sam
More informationInformation Sciences Institute University of Southern California Los Angeles, CA 90292 {annc, carl}@isi.edu
_ Data Management and Transfer in High-Performance Computational Grid Environments Bill Allcock 1 Joe Bester 1 John Bresnahan 1 Ann L. Chervenak 2 Ian Foster 1,3 Carl Kesselman 2 Sam Meder 1 Veronika Nefedova
More informationGlobus Striped GridFTP Framework and Server. Raj Kettimuthu, ANL and U. Chicago
Globus Striped GridFTP Framework and Server Raj Kettimuthu, ANL and U. Chicago Outline Introduction Features Motivation Architecture Globus XIO Experimental Results 3 August 2005 The Ohio State University
More informationWeb Service Robust GridFTP
Web Service Robust GridFTP Sang Lim, Geoffrey Fox, Shrideep Pallickara and Marlon Pierce Community Grid Labs, Indiana University 501 N. Morton St. Suite 224 Bloomington, IN 47404 {sblim, gcf, spallick,
More informationThe Data Grid: Towards an Architecture for Distributed Management and Analysis of Large Scientific Datasets
The Data Grid: Towards an Architecture for Distributed Management and Analysis of Large Scientific Datasets!! Large data collections appear in many scientific domains like climate studies.!! Users and
More informationGridFTP GUI: An Easy and Efficient Way to Transfer Data in Grid
GridFTP GUI: An Easy and Efficient Way to Transfer Data in Grid Wantao Liu, 1,2 Rajkumar Kettimuthu, 3,4 Brian Tieman, 5 Ravi Madduri, 3,4 Bo Li, 1 Ian Foster 2,3,4 1 School of Computer Science and Engineering,
More informationThe Data Grid: Towards an Architecture for the Distributed Management and Analysis of Large Scientific Datasets
The Data Grid: Towards an Architecture for the Distributed Management and Analysis of Large Scientific Datasets Ann Chervenak Ian Foster $+ Carl Kesselman Charles Salisbury $ Steven Tuecke $ Information
More informationComparisons between HTCP and GridFTP over file transfer
Comparisons between HTCP and GridFTP over file transfer Andrew McNab and Yibiao Li Abstract: A comparison between GridFTP [1] and HTCP [2] protocols on file transfer speed is given here, based on experimental
More informationData Grids. Lidan Wang April 5, 2007
Data Grids Lidan Wang April 5, 2007 Outline Data-intensive applications Challenges in data access, integration and management in Grid setting Grid services for these data-intensive application Architectural
More informationGridFTP GUI: An Easy and Efficient Way to Transfer Data in Grid
GridFTP GUI: An Easy and Efficient Way to Transfer Data in Grid Wantao Liu 1,2 Raj Kettimuthu 2,3, Brian Tieman 3, Ravi Madduri 2,3, Bo Li 1, and Ian Foster 2,3 1 Beihang University, Beijing, China 2 The
More informationDataMover: Robust Terabyte-Scale Multi-file Replication over Wide-Area Networks
DataMover: Robust Terabyte-Scale Multi-file Replication over Wide-Area Networks Alex Sim, Junmin Gu, Arie Shoshani, Vijaya Natarajan Lawrence Berkeley National Laboratory (asim, jgu, shoshani, vnatarajan)@lbl.gov
More informationWeb Service Based Data Management for Grid Applications
Web Service Based Data Management for Grid Applications T. Boehm Zuse-Institute Berlin (ZIB), Berlin, Germany Abstract Web Services play an important role in providing an interface between end user applications
More informationA High-Performance Virtual Storage System for Taiwan UniGrid
Journal of Information Technology and Applications Vol. 1 No. 4 March, 2007, pp. 231-238 A High-Performance Virtual Storage System for Taiwan UniGrid Chien-Min Wang; Chun-Chen Hsu and Jan-Jan Wu Institute
More informationGridCopy: Moving Data Fast on the Grid
GridCopy: Moving Data Fast on the Grid Rajkumar Kettimuthu 1,2, William Allcock 1,2, Lee Liming 1,2 John-Paul Navarro 1,2, Ian Foster 1,2,3 1 Mathematics and Computer Science Division Argonne National
More informationConcepts and Architecture of the Grid. Summary of Grid 2, Chapter 4
Concepts and Architecture of the Grid Summary of Grid 2, Chapter 4 Concepts of Grid Mantra: Coordinated resource sharing and problem solving in dynamic, multi-institutional virtual organizations Allows
More informationA Survey Study on Monitoring Service for Grid
A Survey Study on Monitoring Service for Grid Erkang You erkyou@indiana.edu ABSTRACT Grid is a distributed system that integrates heterogeneous systems into a single transparent computer, aiming to provide
More informationA Tutorial on Configuring and Deploying GridFTP for Managing Data Movement in Grid/HPC Environments
A Tutorial on Configuring and Deploying GridFTP for Managing Data Movement in Grid/HPC Environments John Bresnahan Michael Link Rajkumar Kettimuthu Dan Fraser Argonne National Laboratory University of
More informationGlobus XIO Pipe Open Driver: Enabling GridFTP to Leverage Standard Unix Tools
Globus XIO Pipe Open Driver: Enabling GridFTP to Leverage Standard Unix Tools Rajkumar Kettimuthu 1. Steven Link 2. John Bresnahan 1. Michael Link 1. Ian Foster 1,3 1 Computation Institute 2 Department
More informationAn approach to grid scheduling by using Condor-G Matchmaking mechanism
An approach to grid scheduling by using Condor-G Matchmaking mechanism E. Imamagic, B. Radic, D. Dobrenic University Computing Centre, University of Zagreb, Croatia {emir.imamagic, branimir.radic, dobrisa.dobrenic}@srce.hr
More informationA File Transfer Component for Grids
A File Transfer Component for Grids Gregor von Laszewski Mathematics and Computer Science Division, Argonne National Laboratory Argonne, Il 60440, U.S.A. Alunkal Beulah Kurian Mathematics and Computer
More informationMonitoring Data Archives for Grid Environments
Monitoring Data Archives for Grid Environments Jason Lee, Dan Gunter, Martin Stoufer, Brian Tierney Lawrence Berkeley National Laboratory Abstract Developers and users of high-performance distributed systems
More informationGridFTP: Protocol Extensions to FTP for the Grid
Page 1 of 37 GridFTP: Protocol Extensions to FTP for the Grid Status of this Memo This document is an Global Grid Forum Draft and is in full conformance with all provisions of?. Conventions used in this
More informationData Management in an International Data Grid Project. Timur Chabuk 04/09/2007
Data Management in an International Data Grid Project Timur Chabuk 04/09/2007 Intro LHC opened in 2005 several Petabytes of data per year data created at CERN distributed to Regional Centers all over the
More informationGridFTP: Protocol Extensions to FTP for the Grid
Expires: August 2001 Page 1 of 21 GridFTP: Protocol Extensions to FTP for the Grid 1. Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10
More informationGrid Technology and Information Management for Command and Control
Grid Technology and Information Management for Command and Control Dr. Scott E. Spetka Dr. George O. Ramseyer* Dr. Richard W. Linderman* ITT Industries Advanced Engineering and Sciences SUNY Institute
More informationGrid Data Management. Raj Kettimuthu
Grid Data Management Raj Kettimuthu Data Management Distributed community of users need to access and analyze large amounts of data Fusion community s International ITER project Requirement arises in both
More informationGrid Scheduling Dictionary of Terms and Keywords
Grid Scheduling Dictionary Working Group M. Roehrig, Sandia National Laboratories W. Ziegler, Fraunhofer-Institute for Algorithms and Scientific Computing Document: Category: Informational June 2002 Status
More informationFile and Object Replication in Data Grids
File and Object Replication in Data Grids Heinz Stockinger 1,2, Asad Samar 3, Bill Allcock 4, Ian Foster 4,5, Koen Holtman 3, Brian Tierney 1,6 1) CERN, European Organization for Nuclear Research, CH-1211
More informationArgonne National Laboratory April 2003 Revised April 2003. GridFTP: Protocol Extensions to FTP for the Grid
GWD-R (Recommendation) W. Allcock, Editor Argonne National Laboratory April 2003 Revised April 2003 GridFTP: Protocol Extensions to FTP for the Grid Status of this Memo This document is a Global Grid Forum
More informationXSEDE Service Provider Software and Services Baseline. September 24, 2015 Version 1.2
XSEDE Service Provider Software and Services Baseline September 24, 2015 Version 1.2 i TABLE OF CONTENTS XSEDE Production Baseline: Service Provider Software and Services... i A. Document History... A-
More informationDeploying a distributed data storage system on the UK National Grid Service using federated SRB
Deploying a distributed data storage system on the UK National Grid Service using federated SRB Manandhar A.S., Kleese K., Berrisford P., Brown G.D. CCLRC e-science Center Abstract As Grid enabled applications
More informationArchiving, Indexing and Accessing Web Materials: Solutions for large amounts of data
Archiving, Indexing and Accessing Web Materials: Solutions for large amounts of data David Minor 1, Reagan Moore 2, Bing Zhu, Charles Cowart 4 1. (88)4-104 minor@sdsc.edu San Diego Supercomputer Center
More informationTHE CCLRC DATA PORTAL
THE CCLRC DATA PORTAL Glen Drinkwater, Shoaib Sufi CCLRC Daresbury Laboratory, Daresbury, Warrington, Cheshire, WA4 4AD, UK. E-mail: g.j.drinkwater@dl.ac.uk, s.a.sufi@dl.ac.uk Abstract: The project aims
More informationApplied Techniques for High Bandwidth Data Transfers across Wide Area Networks
Applied Techniques for High Bandwidth Data Transfers across Wide Area Networks Jason Lee, Dan Gunter, Brian Tierney Computing Sciences Directorate Lawrence Berkeley National Laboratory University of California,
More informationGrid Computing Research
Grid Computing Research Ian Foster Mathematics and Computer Science Division Argonne National Laboratory and Department of Computer Science The University of Chicago Overview Grid computing & its importance
More informationGlobus Toolkit: Authentication and Credential Translation
Globus Toolkit: Authentication and Credential Translation JET Workshop, April 14, 2004 Frank Siebenlist franks@mcs.anl.gov http://www.globus.org/ Copyright (c) 2002 University of Chicago and The University
More informationHDF5-iRODS Project. August 20, 2008
P A G E 1 HDF5-iRODS Project Final report Peter Cao The HDF Group 1901 S. First Street, Suite C-2 Champaign, IL 61820 xcao@hdfgroup.org Mike Wan San Diego Supercomputer Center University of California
More informationDFSgc. Distributed File System for Multipurpose Grid Applications and Cloud Computing
DFSgc Distributed File System for Multipurpose Grid Applications and Cloud Computing Introduction to DFSgc. Motivation: Grid Computing currently needs support for managing huge quantities of storage. Lacks
More informationUsing the Grid for the interactive workflow management in biomedicine. Andrea Schenone BIOLAB DIST University of Genova
Using the Grid for the interactive workflow management in biomedicine Andrea Schenone BIOLAB DIST University of Genova overview background requirements solution case study results background A multilevel
More informationDATABASES AND THE GRID
DATABASES AND THE GRID Paul Watson Department of Computing Science, University of Newcastle, Newcastle-upon-Tyne, NE1 7RU, UK e-mail: Paul.Watson@newcastle.ac.uk Telephone: +44 191 222 7653 Fax: +44 191
More informationUsing BlobSeer Data Sharing Platform for Cloud Virtual Machine Repository
Using BlobSeer Data Sharing Platform for Cloud Virtual Machine Repository Master Thesis Tuan-Viet DINH vdinh@irisa.fr Supervisors: Gabriel Antoniu, Luc Bougé Gabriel.Antoniu@irisa.fr Luc.Bouge@bretagne.ens-cachan.fr
More informationGrid Computing @ Sun Carlo Nardone. Technical Systems Ambassador GSO Client Solutions
Grid Computing @ Sun Carlo Nardone Technical Systems Ambassador GSO Client Solutions Phases of Grid Computing Cluster Grids Single user community Single organization Campus Grids Multiple user communities
More informationThe THREDDS Data Repository: for Long Term Data Storage and Access
8B.7 The THREDDS Data Repository: for Long Term Data Storage and Access Anne Wilson, Thomas Baltzer, John Caron Unidata Program Center, UCAR, Boulder, CO 1 INTRODUCTION In order to better manage ever increasing
More informationFedora Distributed data management (SI1)
Fedora Distributed data management (SI1) Mohamed Rafi DART UQ Outline of Work Package To enable Fedora to natively handle large datasets. Explore SRB integration at the storage level of the repository
More informationThe glite File Transfer Service
The glite File Transfer Service Peter Kunszt Paolo Badino Ricardo Brito da Rocha James Casey Ákos Frohner Gavin McCance CERN, IT Department 1211 Geneva 23, Switzerland Abstract Transferring data reliably
More informationHDFS Architecture Guide
by Dhruba Borthakur Table of contents 1 Introduction... 3 2 Assumptions and Goals... 3 2.1 Hardware Failure... 3 2.2 Streaming Data Access...3 2.3 Large Data Sets... 3 2.4 Simple Coherency Model...3 2.5
More informationAn Online Credential Repository for the Grid: MyProxy
An Online Credential Repository for the Grid: MyProxy Jason Novotny Lawrence Berkeley Laboratory JDNovotny@lbl.gov Steven Tuecke Mathematics and Computer Science Division Argonne National Laboratory tuecke@mcs.anl.gov
More informationImplementing Network Attached Storage. Ken Fallon Bill Bullers Impactdata
Implementing Network Attached Storage Ken Fallon Bill Bullers Impactdata Abstract The Network Peripheral Adapter (NPA) is an intelligent controller and optimized file server that enables network-attached
More informationGASS: A Data Movement and Access Service for Wide Area Computing Systems
GASS: A Data Movement and Access Service for Wide Area Computing Systems Joseph Bester Ian Foster Carl Kesselman Jean Tedesco Steven Tuecke Abstract In wide area computing, programs frequently execute
More information16th International Conference on Control Systems and Computer Science (CSCS16 07)
16th International Conference on Control Systems and Computer Science (CSCS16 07) TOWARDS AN IO INTENSIVE GRID APPLICATION INSTRUMENTATION IN MEDIOGRID Dacian Tudor 1, Florin Pop 2, Valentin Cristea 2,
More informationGlobus Toolkit Firewall Requirements. Abstract
Globus Toolkit Firewall Requirements Version 9 10/31/06 Von Welch, NCSA/U. of Illinois vwelch@ncsa.uiuc.edu Abstract This document provides requirements and guidance to firewall administrators at sites
More informationUDT as an Alternative Transport Protocol for GridFTP
UDT as an Alternative Transport Protocol for GridFTP John Bresnahan, 1,2,3 Michael Link, 1,2 Rajkumar Kettimuthu, 1,2 and Ian Foster 1,2,3 1 Mathematics and Computer Science Division, Argonne National
More informationCluster, Grid, Cloud Concepts
Cluster, Grid, Cloud Concepts Kalaiselvan.K Contents Section 1: Cluster Section 2: Grid Section 3: Cloud Cluster An Overview Need for a Cluster Cluster categorizations A computer cluster is a group of
More informationTHE EXPAND PARALLEL FILE SYSTEM A FILE SYSTEM FOR CLUSTER AND GRID COMPUTING. José Daniel García Sánchez ARCOS Group University Carlos III of Madrid
THE EXPAND PARALLEL FILE SYSTEM A FILE SYSTEM FOR CLUSTER AND GRID COMPUTING José Daniel García Sánchez ARCOS Group University Carlos III of Madrid Contents 2 The ARCOS Group. Expand motivation. Expand
More informationUsing Globus Toolkit
Using Globus Toolkit G. Poghosyan & D. Nilsen GridKa School 11-15 September 2006 Basic Grid Services in GT Security Services GSI (Grid Security Infrastructure) Data Services GridFTP RFT (Reliable File
More informationGrid Computing: A Ten Years Look Back. María S. Pérez Facultad de Informática Universidad Politécnica de Madrid mperez@fi.upm.es
Grid Computing: A Ten Years Look Back María S. Pérez Facultad de Informática Universidad Politécnica de Madrid mperez@fi.upm.es Outline Challenges not yet solved in computing The parents of grid Computing
More informationIBM Solutions Grid for Business Partners Helping IBM Business Partners to Grid-enable applications for the next phase of e-business on demand
PartnerWorld Developers IBM Solutions Grid for Business Partners Helping IBM Business Partners to Grid-enable applications for the next phase of e-business on demand 2 Introducing the IBM Solutions Grid
More informationz/os Firewall Technology Overview
z/os Firewall Technology Overview Mary Sweat E - Mail: sweatm@us.ibm.com Washington System Center OS/390 Firewall/VPN 1 Firewall Technologies Tools Included with the OS/390 Security Server Configuration
More informationStorage Resource Managers: Middleware Components for Grid Storage
Storage Resource Managers: Middleware Components for Grid Storage Arie Shoshani, Alex Sim, Junmin Gu Lawrence Berkeley National Laboratory Berkeley, California 94720 {shoshani, asim, jgu}@lbl.gov tel +1-510-486-5171
More informationDiagram 1: Islands of storage across a digital broadcast workflow
XOR MEDIA CLOUD AQUA Big Data and Traditional Storage The era of big data imposes new challenges on the storage technology industry. As companies accumulate massive amounts of data from video, sound, database,
More informationVirtualization 101: Technologies, Benefits, and Challenges. A White Paper by Andi Mann, EMA Senior Analyst August 2006
Virtualization 101: Technologies, Benefits, and Challenges A White Paper by Andi Mann, EMA Senior Analyst August 2006 Table of Contents Introduction...1 What is Virtualization?...1 The Different Types
More informationStorage Networking Overview
Networking Overview iscsi Attached LAN Networking SAN NAS Gateway NAS Attached SAN Attached IBM Total Module Flow Business Challenges Networking Trends and Directions What is Networking? Technological
More informationTowards an E-Governance Grid for India (E-GGI): An Architectural Framework for Citizen Services Delivery
Towards an E-Governance Grid for India (E-GGI): An Architectural Framework for Citizen Services Delivery C. S. R. Prabhu 1 ABSTRACT The National e-governance Plan (NeGP) proposes citizen service delivery
More informationGRIP:Creating Interoperability between Grids
GRIP:Creating Interoperability between Grids Philipp Wieder, Dietmar Erwin, Roger Menday Research Centre Jülich EuroGrid Workshop Cracow, October 29, 2003 Contents Motivation Software Base at a Glance
More informationData Management using irods
Data Management using irods Fundamentals of Data Management September 2014 Albert Heyrovsky Applications Developer, EPCC a.heyrovsky@epcc.ed.ac.uk 2 Course outline Why talk about irods? What is irods?
More informationConcepts and Architecture of Grid Computing. Advanced Topics Spring 2008 Prof. Robert van Engelen
Concepts and Architecture of Grid Computing Advanced Topics Spring 2008 Prof. Robert van Engelen Overview Grid users: who are they? Concept of the Grid Challenges for the Grid Evolution of Grid systems
More informationGlobus for Data Management
Globus for Data Management Computation Institute Rachana Ananthakrishnan (ranantha@uchicago.edu) Data Management Challenges Transfers often take longer than expected based on available network capacities
More informationMonitoring Clusters and Grids
JENNIFER M. SCHOPF AND BEN CLIFFORD Monitoring Clusters and Grids One of the first questions anyone asks when setting up a cluster or a Grid is, How is it running? is inquiry is usually followed by the
More informationIntroduction to Arvados. A Curoverse White Paper
Introduction to Arvados A Curoverse White Paper Contents Arvados in a Nutshell... 4 Why Teams Choose Arvados... 4 The Technical Architecture... 6 System Capabilities... 7 Commitment to Open Source... 12
More informationFile Transfer Protocol (FTP) & SSH
http://xkcd.com/949/ File Transfer Protocol (FTP) & SSH Computer Networking: A Top Down Approach 6 th edition Jim Kurose, Keith Ross Some materials copyright 1996-2012 Addison-Wesley J.F Kurose and K.W.
More informationHow To Manage A Monitoring Sensor Management System For A Grid
A Monitoring Sensor Management System for Grid Environments Brian Tierney, Brian Crowley, Dan Gunter, Mason Holding, Jason Lee, Mary Thompson Computing Sciences Directorate Lawrence Berkeley National Laboratory
More informationService Virtualization: Managing Change in a Service-Oriented Architecture
Service Virtualization: Managing Change in a Service-Oriented Architecture Abstract Load balancers, name servers (for example, Domain Name System [DNS]), and stock brokerage services are examples of virtual
More informationMIGRATING DESKTOP AND ROAMING ACCESS. Migrating Desktop and Roaming Access Whitepaper
Migrating Desktop and Roaming Access Whitepaper Poznan Supercomputing and Networking Center Noskowskiego 12/14 61-704 Poznan, POLAND 2004, April white-paper-md-ras.doc 1/11 1 Product overview In this whitepaper
More informationResource Management on Computational Grids
Univeristà Ca Foscari, Venezia http://www.dsi.unive.it Resource Management on Computational Grids Paolo Palmerini Dottorato di ricerca di Informatica (anno I, ciclo II) email: palmeri@dsi.unive.it 1/29
More informationIntroduction. MCSN N. Tonellotto Complements of Distributed Enabling Platforms
Introduction 1 Distributed relating to a computer network in which at least some of the processing is done by the individual computers and information is shared by and often stored at the computers Enabling
More informationArchival Storage Systems and Data Moveability
Configuring and Tuning Archival Storage Systems Reagan Moore, Joseph Lopez, Charles Lofton, Wayne Schroeder, George Kremenek San Diego Supercomputer Center Michael Gleicher Gleicher Enterprises, LLC Abstract
More informationData Movement and Storage. Drew Dolgert and previous contributors
Data Movement and Storage Drew Dolgert and previous contributors Data Intensive Computing Location Viewing Manipulation Storage Movement Sharing Interpretation $HOME $WORK $SCRATCH 72 is a Lot, Right?
More informationNetwork Data Management Protocol (NDMP) White Paper
Network Data Management Protocol (NDMP) White Paper Summary What is the primary goal of enterprise storage management? To back up and restore information in an intelligent, secure, timely, cost-effective
More informationA Next-Generation Analytics Ecosystem for Big Data. Colin White, BI Research September 2012 Sponsored by ParAccel
A Next-Generation Analytics Ecosystem for Big Data Colin White, BI Research September 2012 Sponsored by ParAccel BIG DATA IS BIG NEWS The value of big data lies in the business analytics that can be generated
More informationMonitoring Data Archives for Grid Environments
Monitoring Data Archives for Grid Environments Jason Lee, Dan Gunter, Martin Stoufer, Brian Tierney Lawrence Berkeley National Laboratory Abstract Developers and users of high-performance distributed systems
More informationNETWORK ATTACHED STORAGE DIFFERENT FROM TRADITIONAL FILE SERVERS & IMPLEMENTATION OF WINDOWS BASED NAS
INTERNATIONAL International Journal of Computer JOURNAL Engineering OF COMPUTER and Technology (IJCET), ENGINEERING ISSN 0976-6367(Print), ISSN 0976 & 6375(Online) TECHNOLOGY Volume 4, Issue (IJCET) 3,
More informationAttaching Cloud Storage to a Campus Grid Using Parrot, Chirp, and Hadoop
Attaching Cloud Storage to a Campus Grid Using Parrot, Chirp, and Hadoop Patrick Donnelly, Peter Bui, Douglas Thain Computer Science and Engineering University of Notre Dame pdonnel3@nd.edu pbui@nd.edu
More informationA Web Services Data Analysis Grid *
A Web Services Data Analysis Grid * William A. Watson III, Ian Bird, Jie Chen, Bryan Hess, Andy Kowalski, Ying Chen Thomas Jefferson National Accelerator Facility 12000 Jefferson Av, Newport News, VA 23606,
More informationWorkload Characterization and Analysis of Storage and Bandwidth Needs of LEAD Workspace
Workload Characterization and Analysis of Storage and Bandwidth Needs of LEAD Workspace Beth Plale Indiana University plale@cs.indiana.edu LEAD TR 001, V3.0 V3.0 dated January 24, 2007 V2.0 dated August
More informationWhite Paper. Requirements of Network Virtualization
White Paper on Requirements of Network Virtualization INDEX 1. Introduction 2. Architecture of Network Virtualization 3. Requirements for Network virtualization 3.1. Isolation 3.2. Network abstraction
More informationExploration of adaptive network transfer for 100 Gbps networks Climate100: Scaling the Earth System Grid to 100Gbps Network
Exploration of adaptive network transfer for 100 Gbps networks Climate100: Scaling the Earth System Grid to 100Gbps Network February 1, 2012 Project period of April 1, 2011 through December 31, 2011 Principal
More informationData Management in an International Data Grid Project
Data Management in an International Data Grid Project Wolfgang Hoschek 1,3, Javier Jaen-Martinez 1, Asad Samar 1,4, Heinz Stockinger 1,2, and Kurt Stockinger 1,2 1 CERN, European Organization for Nuclear
More informationA Distributed Architecture for Multi-dimensional Indexing and Data Retrieval in Grid Environments
A Distributed Architecture for Multi-dimensional Indexing and Data Retrieval in Grid Environments Athanasia Asiki, Katerina Doka, Ioannis Konstantinou, Antonis Zissimos and Nectarios Koziris National Technical
More informationData Management System for grid and portal services
Data Management System for grid and portal services Piotr Grzybowski 1, Cezary Mazurek 1, Paweł Spychała 1, Marcin Wolski 1 1 Poznan Supercomputing and Networking Center, ul. Noskowskiego 10, 61-704 Poznan,
More informationKNOWLEDGE GRID An Architecture for Distributed Knowledge Discovery
KNOWLEDGE GRID An Architecture for Distributed Knowledge Discovery Mario Cannataro 1 and Domenico Talia 2 1 ICAR-CNR 2 DEIS Via P. Bucci, Cubo 41-C University of Calabria 87036 Rende (CS) Via P. Bucci,
More informationNext-Generation Federal Data Center Architecture
Next-Generation Federal Data Center Architecture Introduction The United States government s E-Gov initiative established the Federal Enterprise Architecture (FEA) program to build a business-driven information
More information2 Transport-level and Message-level Security
Globus Toolkit Version 4 Grid Security Infrastructure: A Standards Perspective The Globus Security Team 1 Version 4 updated September 12, 2005 Abstract This document provides an overview of the Grid Security
More informationChapter 3. Database Environment - Objectives. Multi-user DBMS Architectures. Teleprocessing. File-Server
Chapter 3 Database Architectures and the Web Transparencies Database Environment - Objectives The meaning of the client server architecture and the advantages of this type of architecture for a DBMS. The
More informationAn Overview of Parallelism Exploitation and Cross-layer Optimization for Big Data Transfers
1 An Overview of Parallelism Exploitation and Cross-layer Optimization for Big Data Transfers Eun-Sung Jung, Member, IEEE, and Rajkumar Kettimuthu, Senior Member, IEEE Abstract The data produced by sensors
More informationIntegration of Network Performance Monitoring Data at FTS3
Integration of Network Performance Monitoring Data at FTS3 July-August 2013 Author: Rocío Rama Ballesteros Supervisor(s): Michail Salichos Alejandro Álvarez CERN openlab Summer Student Report 2013 Project
More informationOS/390 Firewall Technology Overview
OS/390 Firewall Technology Overview Washington System Center Mary Sweat E - Mail: sweatm@us.ibm.com Agenda Basic Firewall strategies and design Hardware requirements Software requirements Components of
More informationMoving Grid Systems into the IPv6 Era
Moving Grid Systems into the IPv6 Era Sheng JIANG, Piers O Hanlon, Peter Kirstein Department of Computer Science University College London Gower Street, WC1E 6BT London, United Kingdom {S.Jiang, P.Ohanlon,
More informationOS/390 Firewall Technology Overview
OS/390 Firewall Technology Overview Mary Sweat E - Mail: sweatm@us.ibm.com Washington System Center OS/390 Firewall/VPN 1 Agenda OS/390 Firewall OS/390 Firewall Features Hardware requirements Software
More informationCHAPTER 2 MODELLING FOR DISTRIBUTED NETWORK SYSTEMS: THE CLIENT- SERVER MODEL
CHAPTER 2 MODELLING FOR DISTRIBUTED NETWORK SYSTEMS: THE CLIENT- SERVER MODEL This chapter is to introduce the client-server model and its role in the development of distributed network systems. The chapter
More informationIn Memory Accelerator for MongoDB
In Memory Accelerator for MongoDB Yakov Zhdanov, Director R&D GridGain Systems GridGain: In Memory Computing Leader 5 years in production 100s of customers & users Starts every 10 secs worldwide Over 15,000,000
More information