Client/Server Grid applications to manage complex workflows

Transcription

1 Client/Server Grid applications to manage complex workflows Filippo Spiga* on behalf of CRAB development team * INFN Milano Bicocca (IT)

2 Outline Science Gateways and Client/Server computing Client/server approach for CMS distributed analysis Usage statistics and trends Considerations, remarks and future works 2

3 Science Gateways & Client/Server Computing A Science Gateway is a community-developed set of tools, applications, and data collections that are integrated via a portal or a suite of applications. The Client/Server Computing is a distributed application architecture that partitions tasks or work loads between service providers (servers) and service requesters (clients). 3

4 Science Gateways & Client/Server Computing Science gateways and client/server applications share the main fundamental ideas address the same purposes are realized using different technologies are deployed in different ways interact with users through different interfaces The choice between them will be made considering the target community (both users and developers) 4

5 Motivations & Advantages (for users) In the context of HEP computing, the client/server applications can automate (as much as possible) complex analysis workflows allow more advanced job monitoring with centralized front ends hide configuration complexity on heterogeneous grid environment hide implementation details to the users automatic resubmission in case of failure allow (strong) centralized policies 5

6 Motivations & Advantages (for operators and developers) In the context of HEP computing, the client/server applications can extend/integrate the functionalities of the middleware help to manage centrally updates improve support to users by operators avoid destructive congestions interacting with the grid improve the scalability of the entire computing model 6

7 The experience with CRAB tool in CMS CRAB, a Python tool intended to create, submit and manage CMS analysis jobs in a Grid environment. A single tool addresses different applications/purposes: CMSSW analysis over the Grid private MC production complex workflow automation Started as standalone application it is evolved to a client/server application (in production since July 2007) 7

8 Standalone vs Client-Server (workflow point of view) UI C R A B CRAB DatasetPath LFNs, #events, fileblocks DBS Dataset Discovery CRAB server Proxy UserBox CRAB Client datasetpath DBS Output jdl, job, RB/WMS jdl, job CE WN... Fileblocks SEs Data SE PhEDEx DLS (DLI interface) Dataset Location trivial file catalog LFN!> PFN File Location BOSS DB Output gsiftp server Job jdl, CE WN... Jdl, Job direct submission available jdl RB LFN, #events, fileblocks Data fileblocks SEs fileblocks SEs SE File Location Dataset Discovery PhEDEx DLS (DLI enabled, LFC based Dataset Location trivial file catalog Output (also to remote SE) Output 8

9 Server architecture Agent-based model USER (BROWSER, MAILBOX, USER INTERFACE) Publish&Subscribe message service Most components are multi-threading Transparent support to multiple storage systems (SEAPI) and multiple middlewares (BossLite) TASK TRACKING ADMIN CONTROL ERROR HANDLER JOB KILLER COMMAND MANAGER JOB TRACKING HTTP FRONTEND MYSQL DATABASE (BOSSLITE) MIDDLEWARE GET OUTPUT NOTIFI- CATION CRAB JOB CREATOR TASK LIFE MANAGER TASK REGISTER CRAB SERVER WORKER STORAGE MYPROXY 9

10 Components interaction example (submission) CLIENT gsoap SERVER SQLITE DB -submit COMMAND MANAGER If a component/thread crashes, the system continues to work GSIFTP Storage Element TASK REGISTER CRAB SERVER WORKER MySQL DATABASE GRID Other services provided by the server will be not compromised At worst, smooth degradation of QoS 10

11 CRAB deployment on Grid infrastructure CRAB deployment strategy for CMS enforce fault-tolerance avoid single point congestion glite is the main middleware used periodic challenges are performed to test and stress the infrastructure Different CRAB-AS in production now located at CERN (CH), UCSD (US), BARI (IT) each physic groups can deploy it s own unofficial server Analysis Operators team (AnaOps) manages and monitors the services 11

12 Statistics: starting point Source of statistics: CMS Dashboard Focus on comparison between standalone and client/ server approaches trend analysis from Jan 2008 to Feb 2010 Statistic is not perfect sometimes testing jobs submitted by CRAB developers are counted as analysis jobs Only analysis jobs on T2 level were considered 12

13 CRAB Standalone vs Server ( ) 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% SERVER DIRECT SERV; 23% DIRECT; 77% SERV; 47% DIRECT; 53% Source: CMS Dashboard, 27/03/

14 CRAB Standalone vs Server (last 6 months, Sep 09 - Feb 10) 54% of submissions were made through the server Source: CMS Dashboard, 27/03/

15 CRAB Standalone vs Server (2009) Restart of LHC Number of distinct CRAB users: 150~200/day Max distinct users peak reached: ~250 Source: CMS Dashboard, ~02/

16 Considerations & Remarks Architectural aspects of the software are the key factors to reach success good performance achieved good stability and reliability Inside CMS community, the adoption of client/server approach is increasing good feedback by users good quality of service reached The only (relevant but known) problems are related to data movement stage-out remains a critical operation across the Grid, influenced by several factors 16

17 Improvements & Future works Consolidate the actual architecture change the interaction with glite middleware (CLI instead of API) better strategies for stage-out operations maintain the best interoperability for all middlewares supported reduce the number of jobs aborted improvements for intelligent resubmission New framework for all CMS computing: WMCore one application targets the online (T0), the official production (T1) and the user analyses (T2) communications based on RESTful web services too young to go to production (expected for ) 17

18 THANK YOU FOR YOUR ATTENTION Questions? Special thanks to Daniele Spiga (CRAB), Fabio Farina (CRAB), Mattia Cinquilli (CRAB) and Julia Andreeva (CMS Dashboard) 18

19 The CRAB history (backup slide) CRAB_2_X_X (10/2007) CRAB_2_7_1 (03/2010) CRAB_1_X_X 2004/ CRABSERVER_0_0_1 (07/2007) CRABSERVER_1_0_X (05/2008) CRABSERVER_1_1_0 (11/2009) CRABSERVER_1_1_1 (03/2010) 19