Using ssh as portal CHEP The CMS CRAB over glideinwms experience

Transcription

1 CHEP 2013 Using ssh as portal The CMS CRAB over glideinwms experience by I Sfiligoi 1, S Belforte 2, J Letts 1, T Martin 1, M D Saiz Santos 1 and F Fanzago 3 1 University of California San Diego 2 Università e INFN di Trieste 3 Università e INFN di Padova CHEP 2013 Using SSH as portal for CMS 1

2 Agenda How we got there? What we did? How it worked out? CHEP 2013 Using SSH as portal for CMS 2

3 CMS AnaOps CMS is one of the 4 experiments at the Large Hadron Collider CMS Analysis Operations (AnaOps) is a computing task focused on the operational aspects of enabling physics data analysis i.e. provides the computing infrastructure used by the actual physicists CHEP 2013 Using SSH as portal for CMS 3

4 CRAB CMS AnaOps submission tool CMS AnaOps has been using CRAB as the user interface to the Grid since 2004 Initially was a purely client interface CRAB basically just a fancy wrapper So the Client machine had to have the full Grid stack to function CHEP 2013 Using SSH as portal for CMS 4

5 CRAB2 Server CRAB2 Server introduced in 2007 Client does not need the Grid stack anymore CHEP 2013 Using SSH as portal for CMS 5

6 glideinwms In 2009 CMS added glideinwms to the list of supported middleware glideinwms, based on HTCondor, does not support remote job submission CRAB2 Server had to sit on the same node as (one of) the HTCondor scheduler(s) CHEP 2013 Using SSH as portal for CMS 6

7 glideinwms and CRAB2 Server A schematic view HTCondor Scheduler glideinwms CHEP 2013 Using SSH as portal for CMS 7

8 Operational experience CMS AnaOps experienced a lot of operational issues with CRAB2 Server submitting through glideinwms One major problem being the server losing track of job status Users were complaining a lot CHEP 2013 Using SSH as portal for CMS 8

9 A look at the CRAB2 Server Designed to be very modular and flexible But that made it complex, too CMS decided to give up on fixing it CRAB3 would simply replace it CHEP 2013 Using SSH as portal for CMS 9

10 A look at the CRAB2 Server Designed to be very modular and flexible But that made it complex, too CMS decided to give up on fixing it CRAB3 would simply replace it But by late 2012, CRAB3 still did not exist CHEP 2013 Using SSH as portal for CMS 10

11 What was wrong with CRAB2 Client? The CRAB2 Client never went away A significant fraction of CMS users kept using it after CRAB2 Server was put in production And they were, by and large, liking it It just could not be used to submit to glideinwms Due to lack of (native) remote submission to HTCondor CHEP 2013 Using SSH as portal for CMS 11

12 Why not just use ssh? The major stated benefit of CRAB2 Server was remote submission using a standard interface (there were of course other stated benefits, but they had very little impact when paired with glideinwms) But what users normally use for remote access? SSH So, what if we used ssh in CRAB2 client as well? The gsissh dialect to support x509 proxies CHEP 2013 Using SSH as portal for CMS 12

13 The ssh-enabled CRAB2 Client There are three stages in CRAB2 Client Job submission Includes uploading the input sandbox Job monitoring Output log fetching The Client uses ssh and/or scp to talk to a node running the HTCondor scheduler CHEP 2013 Using SSH as portal for CMS 13

14 The ssh-enabled CRAB2 Client A schematic view Client node crab_submit scp input_sandox ssh condor_submit ssh crab_monitor ssh condor_q HTCondor Scheduler crab_fetch scp output_logs glideinwms CHEP 2013 Using SSH as portal for CMS 14

15 Server side setup gsisshd is a standard package Just installed it from RPM Using lcmaps for authorization With callbacks to GUMS in the USA, and Argus in Europe Each user gets a standard Linux account No technical limits on what the user can run, but we would kill any offenders, if spotted CHEP 2013 Using SSH as portal for CMS 15

16 Users liked it! Took only 4 months for all users to completely switch from Server to ssh-client for glideinwms And another 4 months to get 90% of all work ssh-based Client CHEP 2013 Using SSH as portal for CMS 16

17 Smooth operations We have not seen any major problems with ssh-based CRAB2 client itself Now almost all problems are in the Grid layer Maybe not too surprising CRAB2 Client is really just a fancy wrapper SSH is a very mature tool Operational load now at historical low in spite of increasing usage CHEP 2013 Using SSH as portal for CMS 17

18 Works at scale Essentially ssh-only Essentially ssh-only Includes non-ssh jobs as well Includes non-ssh jobs as well CHEP 2013 Using SSH as portal for CMS 18

19 Hickups during the voyage We occasionally hit a ssh quirk In order to speed up ssh connections, we re-use the ssh control session (-S) If it gets stuck, user needed to manually remove it CRAB2 Client now automatically detects and fixes the problem CHEP 2013 Using SSH as portal for CMS 19

20 Remaining issue The major remaining issue is the distribution of the CRAB2 Client code This still must be installed on the client node Any change must be pushed to ALL the users Making server-side changes challenging CHEP 2013 Using SSH as portal for CMS 20

21 Conclusions Using (gsi)ssh as a portal has proven to be very effective By removing complexity out of the code we have to maintain, life got much easier CHEP 2013 Using SSH as portal for CMS 21

22 Acknowledgments The idea of using ssh came from the RCondor package (see poster #322) This work was partially sponsored by the US National Science Foundation under Grants No. PHY and PHY CHEP 2013 Using SSH as portal for CMS 22