The glite File Transfer Service

Size: px

Start display at page:

Download "The glite File Transfer Service"

Benjamin Hodges
8 years ago
Views:

1 Enabling Grids Enabling for E-sciencE Grids for E-sciencE The glite File Transfer Service Paolo Badino On behalf of the JRA1 Data Management team EGEE User Forum - CERN, 2 Mars

Badino On behalf of the JRA1 Data Management team

2 Outline The glite File Transfer Service in LCG Service Challenge 3 Architecture Administration Security Result Current Work Focus EGEE User Forum: The glite File Transfer Service 2

3 glite File Transfer Service Low Level, point to point data movement service Part of the glite Middleware Asynchronous transfers Based on Globus GridFTP, SRM Retry policies Not just a Reliable Transfer Service EGEE User Forum: The glite File Transfer Service 3

transfers Based on Globus GridFTP, SRM Retry policies Not just a

4 LCG Service Challenges Production Quality Focus on Service: Network service Robust file transfer service Sustainability Heterogeneous Environment Castor2, dcache, DPM Service Challenge 2 Radiant Service Challenge 3 FTS (glite 1.4.1) CMS Phedex EGEE User Forum: The glite File Transfer Service 4

Environment Castor2, dcache, DPM Service Challenge 2 Radiant Service

5 The Key Concept: The Channel Enabling Grids for E-sciencE Logical unit of management Represent a directed network pipe between two sites Mono-directional Independently manageable State Number of streams Number of concurrent transfers Inter-VO scheduling VO share No Routing EGEE User Forum: The glite File Transfer Service 5

Independently manageable State Number of streams Number of concurrent transfers

6 Non-Dedicated Channels Catch-all e.g. CERN-*, *-CERN, *-* Site Grouping Define groups (clouds) based on attributes EGEE User Forum: The glite File Transfer Service 6

7 FTS Architecture FTS Web Service User, Administration and Monitoring Interfaces File Transfer Queue File Transfer Agents VO Agents Channels Agents SOAP API and CLI EGEE User Forum: The glite File Transfer Service 7

Transfer Agents VO Agents Channels Agents SOAP API

8 Transfer Jobs and Files Job Represent the transfer request Identified by a GUID File source-destination file names pair Job States File States EGEE User Forum: The glite File Transfer Service 8

9 Channel Administration Channel has a number of properties State Active Inactive Drain Stopped Halted Number of concurrent files transfers Number of streams VO Share EGEE User Forum: The glite File Transfer Service 9

Number of concurrent files transfers Number of

10 Transfer Type Enabling Grids for E-sciencE SRM Get & Put + 3 rd party GridFTP SRM Copy Timeouts SRM interactions Transfer Transfer Markers TCP Buffer Size Channel Agent Properties EGEE User Forum: The glite File Transfer Service 10

Transfer Transfer Markers TCP Buffer Size Channel Agent

11 Security: Roles Submitter User Administrator Vetoed User Channel Manager VO Manager Gridmap files VOMS roles EGEE User Forum: The glite File Transfer Service 11

12 Security: Delegation Delegated User Credentials retrieved from MyProxy Agents are Authorized Retrievers problematic Delegation PortType EGEE User Forum: The glite File Transfer Service 12

13 Monitoring Provides statistic information based on the FTS behavior and usage Snapshots of number of requests in a given state Average rate and bytes written in a given period Grouped by VO or Channels Channel Monitoring EGEE User Forum: The glite File Transfer Service 13

rate and bytes written in a given period Grouped by VO or Channels

14 FTS Web Interface EGEE User Forum: The glite File Transfer Service 14

15 Retry Logic Simple retry strategy Retry all the files up to 3 times, independently of the error reason Complex retry strategy (glite 1.5) Evaluate the transfer attempts history Resilient to failures in the error handling VO specific retry logic (glite 1.5) EGEE User Forum: The glite File Transfer Service 15

5) Evaluate the transfer attempts history Resilient to failures in the error

16 What SC achieved so far SC3 Rerun (January 2006) All sites achieved target rate 8/11 sites achieved nominal rate EGEE User Forum: The glite File Transfer Service 16

17 Some Numbers Transfer Request in January 2006 (mainly SC3 rerun) ~ Transfer job requests ~ Files > 1 million transfer attempts Job States 5% File States Transfer States 11% 0% 24% 2% 1% 10 % 65% Done Canceled Failed Hold 0% 12% 1% 1% 6% Done Done Canceled Failed Hold 18% 38% 0% 62% Completed Aborted Failed 87% 81% 94% 0% 82% 55% 18% File File Errors 16% 8% 2%0% 1% 0% 23% 20% 57% Agent Destination Source Netw Netw ork ork Transfer Security Consistency 23% 58% 15% Transfer Errors Transfer Errors 0% 0% 0% 24% 18% 62% Agent Agent Destination Destination Source Source Netw ork Network Transfer Transfer Security Security EGEE User Forum: The glite File Transfer Service 17

82% 55% 18% File File Errors 16% 8% 2%0% 1% 0% 23% 20% 57% Agent Destination Source Netw Netw ork ork Transfer Security Consistency 23% 58% 15% Transfer Errors Transfer

18 Current Focus LCG Service Challenge 4 Stability Sustainability Robustness Monitoring Procedures SRM v2 Hooks/Catalog plug-ins EGEE User Forum: The glite File Transfer Service 18

19 Summary glite FTS Reliable Transfer Service Transfer Management Tool LCG Service Challenge 3 Architecture Channels Security Results achieved Current Focus LCG Service Challenge 4 New features EGEE User Forum: The glite File Transfer Service 19

Channels Security Results achieved Current Focus LCG Service

20 Links glite glite JRA1 Data Management Service Challenges Wiki FTS Wiki EGEE User Forum: The glite File Transfer Service 20

cern.ch/twiki/bin/view/egee/fts/ EGEE User Forum: The glite File Transfer

File Transfer Software and Service SC3

File Transfer Software and Service SC3 Gavin McCance JRA1 Data Management Cluster Service Challenge Meeting April 26 2005, Taipei www.eu-egee.org Outline Overview of Components Tier-0 / Tier-1 / Tier-2