Data storage in Jülich

Transcription

1 Data storage in Jülich GLOW Meeting 2012 Bielefeld 19. June 2012 Cristina Manzano

2 Data storage in Jülich changes LOFAR data placement & access policy Pools for the LOFAR groups LOFAR/ops and LOFAR/test. ACLs for controlling the access to the system. Configuration Shut down the 4 pool-only machines not connected to the LOFAR network (10Gig interface). Doors communicate to the pools only through the 10Gig interface. 19. June 2012 GLOW Meeting Bielefeld 2

3 Data storage in Jülich configuration DSC Lofar Net 16.3 K08 FC Pool/Door12 Admin,srm Pool/Door11 Pool/Door10 DownloadServer Pool/Door9 24 x 4TB TSM DMZ JuRoPA 19. June 2012 GLOW Meeting Bielefeld 3

4 Data storage in Jülich configuration Storage Element (SE): 1 admin node 4 pool+door nodes 90 TB disk 1 PB tape Download Server: Username-Password 19. June 2012 GLOW Meeting Bielefeld 4

5 Data storage in Jülich data transfers 19. June 2012 GLOW Meeting Bielefeld 5

6 Data storage in Jülich data transfers 19. June 2012 GLOW Meeting Bielefeld 6

7 Data storage in Jülich data transfers 19. June 2012 GLOW Meeting Bielefeld 7

8 Data storage in Jülich data transfers February TB March TB April TB Total LOFAR data stored currently TB 19. June 2012 GLOW Meeting Bielefeld 8

9 Data storage in Jülich solved issues When the file is really big checksum timeout limit of 4 hours per send request reached transfer fails. Solution: change the checksum policy; checksum is calculated on the fly instead on after the tranfer (csm set policy -onwrite=off -ontransfer=on). The drivers of the 10Gig network cards in the doors had some bug which made the interface die when heavily used. Solution: drivers updated. 19. June 2012 GLOW Meeting Bielefeld 9

10 Data storage in Jülich future work Add funcionalities to the Download Server Authorization against an LDAP service, include SRMLS to check the status of the file, bring online request. Solve the pool selection mechanism issues Update to the new dcache golden release. Set doors in both the private network and the DMZ network Not a default dcache configuration! In contact with the dcache developers and waiting for a solution. 19. June 2012 GLOW Meeting Bielefeld 10

11 Data storage in Jülich Thanks for your attention! Questions? 19. June 2012 GLOW Meeting Bielefeld 11

12 Data storage in Jülich extra slides 19. June 2012 GLOW Meeting Bielefeld 12

13 Data storage in Jülich work done Administration and coordination work LTA notification distribution: mailing list LOFAR LTA operations meeting: Capacity planning versus available capacity and e.g. planned maintenance. Representatives of LOFAR operations, and the Juelich, Amsterdam and Groningen LTA sites. Harm Munk (head of LOFAR Technical Operations) is taking the initiative. LOFAR LTA coordination meeting: Technical matters. Representatives of the different LTA sites. 19. June 2012 GLOW Meeting Bielefeld 13

14 Data storage in Jülich data transfers Performance from the last transfers on 8th of May From lexar001 to Jülich archive Average MBps Max MBps From lexar002 to Jülich archive Average MBps Max MBps 19. June 2012 GLOW Meeting Bielefeld 14

15 Data storage in Jülich data transfers 19. June 2012 GLOW Meeting Bielefeld 15

16 Data storage in Jülich data transfers 19. June 2012 GLOW Meeting Bielefeld 16