Tools for Data Transfer

Transcription

1 Tools for Data Transfer Dr. David Swanson Director, Holland Compu:ng Center University of Nebraska

2 Outline Data Transfer Context Data Transfer Tools Example Cases New Research

3 on- demand c3.xlarge hfps://github.com/djw8605/hccdashboard

4

5 Internet2 via GPN Internet2 via CIC UNL Science DMZ Composit Traffic 100 Gigabit Composit Traffic 100 Gigabit Future Redundant I2 Path (2015+) Holland Computing Center WDM 100 Gigabit Capable West Campus Border Router 100 Gigabit Capable East Campus Border Router Lincoln Core Router Omaha Core 2x 10 Gigabit perfsonar + BRO IDS additions East/West Campus Networks (firewalls + IDS) Omaha HPC Clusters Lincoln Core Switch (CMS and HPC clusters) 10x 10 Gigabit DYNES Equipment Center for Brain Imaging and Behavior Campus Network Researchers 10 Gigabit 4x 10 Gigabit 100 Gigabit perfsonar Bro IDS 10x CMS Data Transfer Nodes

6 Storage Op:ons /home ZFS, quotas, backed up /work Lustre, no quota, purged /shared ZFS, not backed up HDFS (CMS only) Library custodial storage Offsite Tape increased security, per year

7 Data Transfer Tools GridFTP FTW! requires considera:on admin interven:on Also use: hfp, condor file transfer, SCP, winscp Aspera (not using due to nature of network) Have ordered transfer boxes for Globus Online endpoints much more accessible for users

8 Data Movement in CMS Subscription Request PhEDEx Status/Registra%on Hosting Site Storage SRM Grid`p Registra%on DAS/DBS Data Loca:on Authority Transfer Instructions Target Site Agent Nego%a%on SRM Grid`p Target Site Storage Transfers

9 Batch Transfers Data Transfer Op:ons < 100 MB Condor File Transfer [UWisc] < 500 MB hfp caching (squid) < 2 GB GridFTP [Globus] Streaming Transfers small, generic Parrot/Chirp [CCTools, ND] big, HEP xrootd [SLAC]

10

11 Any Data, Any Time, Anywhere NSF- sponsored project aiming to increase data access for HEP While HEP has a long history in bulk data transfers between sites, want to enable direct remote IO to users applica:ons With remote IO, users can access the experiments files in seconds, not hours

12 AAA technology - Xrootd Similar in spirit to a content delivery network, although unique in that the access paferns are inherently cache- unfriendly and the volume of the data (~30PB) allows CMS to run jobs at one site while streaming data from a second site currently plays an integral role in the global CMS compu:ng evolu:on plans

13 AAA technology - Xrootd

14 LARK NSF grant in the CC- NIE program, Lark aims to improve network usage for HTCondor distributed high- throughput compu:ng integrates and tests advanced network technologies (IPv6, perfsonar, network management via SDN), with the goal of bringing them into produc:on at UNL and University of Wisconsin Madison

15 LARK For example, provide each HTCondor batch job with an IP address so the network layer can manage at a per- job, not per- host, level.

16 Credit Due Not possible without help/support from I2, GPN, ITS (Mark Askren, CEO) networking (Kent Christensen) OSG, UW- Madison, ND, Globus NSF Dr. Brian Bockelman, Dr. Adam Caprez, Dr- 2- b Derek Weitzel, Garhan AFebury NU Administra:on, NRI, Holland Founda:on