Enabling Grids Enabling for E-sciencE Grids for E-sciencE The glite File Transfer Service Paolo Badino On behalf of the JRA1 Data Management team EGEE User Forum - CERN, 2 Mars 2006 www.eu-egee.org
Outline The glite File Transfer Service in LCG Service Challenge 3 Architecture Administration Security Result Current Work Focus EGEE User Forum: The glite File Transfer Service 2
glite File Transfer Service Low Level, point to point data movement service Part of the glite Middleware Asynchronous transfers Based on Globus GridFTP, SRM Retry policies Not just a Reliable Transfer Service EGEE User Forum: The glite File Transfer Service 3
LCG Service Challenges Production Quality Focus on Service: Network service Robust file transfer service Sustainability Heterogeneous Environment Castor2, dcache, DPM Service Challenge 2 Radiant Service Challenge 3 FTS (glite 1.4.1) CMS Phedex EGEE User Forum: The glite File Transfer Service 4
The Key Concept: The Channel Enabling Grids for E-sciencE Logical unit of management Represent a directed network pipe between two sites Mono-directional Independently manageable State Number of streams Number of concurrent transfers Inter-VO scheduling VO share No Routing EGEE User Forum: The glite File Transfer Service 5
Non-Dedicated Channels Catch-all e.g. CERN-*, *-CERN, *-* Site Grouping Define groups (clouds) based on attributes EGEE User Forum: The glite File Transfer Service 6
FTS Architecture FTS Web Service User, Administration and Monitoring Interfaces File Transfer Queue File Transfer Agents VO Agents Channels Agents SOAP API and CLI EGEE User Forum: The glite File Transfer Service 7
Transfer Jobs and Files Job Represent the transfer request Identified by a GUID File source-destination file names pair Job States File States EGEE User Forum: The glite File Transfer Service 8
Channel Administration Channel has a number of properties State Active Inactive Drain Stopped Halted Number of concurrent files transfers Number of streams VO Share EGEE User Forum: The glite File Transfer Service 9
Transfer Type Enabling Grids for E-sciencE SRM Get & Put + 3 rd party GridFTP SRM Copy Timeouts SRM interactions Transfer Transfer Markers TCP Buffer Size Channel Agent Properties EGEE User Forum: The glite File Transfer Service 10
Security: Roles Submitter User Administrator Vetoed User Channel Manager VO Manager Gridmap files VOMS roles EGEE User Forum: The glite File Transfer Service 11
Security: Delegation Delegated User Credentials retrieved from MyProxy Agents are Authorized Retrievers problematic Delegation PortType EGEE User Forum: The glite File Transfer Service 12
Monitoring Provides statistic information based on the FTS behavior and usage Snapshots of number of requests in a given state Average rate and bytes written in a given period Grouped by VO or Channels Channel Monitoring EGEE User Forum: The glite File Transfer Service 13
FTS Web Interface EGEE User Forum: The glite File Transfer Service 14
Retry Logic Simple retry strategy Retry all the files up to 3 times, independently of the error reason Complex retry strategy (glite 1.5) Evaluate the transfer attempts history Resilient to failures in the error handling VO specific retry logic (glite 1.5) EGEE User Forum: The glite File Transfer Service 15
What SC achieved so far SC3 Rerun (January 2006) All sites achieved target rate 8/11 sites achieved nominal rate EGEE User Forum: The glite File Transfer Service 16
Some Numbers Transfer Request in January 2006 (mainly SC3 rerun) ~ 16000 Transfer job requests ~ 787000 Files > 1 million transfer attempts Job States 5% File States Transfer States 11% 0% 24% 2% 1% 10 % 65% Done Canceled Failed Hold 0% 12% 1% 1% 6% Done Done Canceled Failed Hold 18% 38% 0% 62% Completed Aborted Failed 87% 81% 94% 0% 82% 55% 18% File File Errors 16% 8% 2%0% 1% 0% 23% 20% 57% Agent Destination Source Netw Netw ork ork Transfer Security Consistency 23% 58% 15% Transfer Errors Transfer Errors 0% 0% 0% 24% 18% 62% Agent Agent Destination Destination Source Source Netw ork Network Transfer Transfer Security Security EGEE User Forum: The glite File Transfer Service 17
Current Focus LCG Service Challenge 4 Stability Sustainability Robustness Monitoring Procedures SRM v2 Hooks/Catalog plug-ins EGEE User Forum: The glite File Transfer Service 18
Summary glite FTS Reliable Transfer Service Transfer Management Tool LCG Service Challenge 3 Architecture Channels Security Results achieved Current Focus LCG Service Challenge 4 New features EGEE User Forum: The glite File Transfer Service 19
Links glite http://www.glite.org/ glite JRA1 Data Management http://egee-jra1-dm.web.cern.ch/egee-jra1-dm/ Service Challenges Wiki https://twiki.cern.ch/twiki/bin/view/lcg/lcgservicechallenges FTS Wiki https://twiki.cern.ch/twiki/bin/view/egee/fts/ EGEE User Forum: The glite File Transfer Service 20