dcache - Managed Storage - LCG Storage Element - HSM optimizer Patrick Fuhrmann, DESY for the dcache Team

Transcription

1 dcache - Managed Storage - LCG Storage Element - HSM optimizer, DESY for the dcache Team dcache is a joint effort between the Deutsches Elektronen Synchrotron (DESY) and the Fermi National Laboratory (FNAL)

2 Support Model dcache Collaboration Fnal DESY CDF Grid KA Karlruhe US CMS Brookhaven LCG Deployment CERN LCG Tier I / II

3 dcache - LCG support model Developers FERMI Developers DESY dcache Deployment (DESY) Ticket System Support Tools Web Pages Downloads Call Center LCG Deployment LCG dcache Evaluation Team Support Tools Web Pages Downloads Call Center Ticket System Grid KA LHC Tier I / II center

4 dcache the Specs

5 Basic dcache System dcache : Managed Disk Storage System Basic Specification Single 'rooted' file system name space tree File system names space view available through an nfs2/3 interface Data is distributed among a huge amount of disk servers. Supports multiple internal and external copies of a single file Supports 'posix like' (authenticated) access as well as various FTP dialects and the Storage Resource Manager Protocol.

6 Basic dcache System Scalability Distributed Movers AND Access Points (Doors) Automatic load balancing using cost metric and inter pool transfers. Pool 2 Pool transfers on pool hot spot detection Handles bunch requests by fast pool selection unit

7 Basic dcache System Configuration Fine grained configuration of pool attraction scheme. (write pools, subnet, directory tree, storage info) Pool to pool transfers on configuration of forbidden transfers Fine grained tuning : Space vs. Mover cost preference Tertiary Storage Manager connectivity Automatic HSM migration and restore HSM dcache interface by script (shell, perl...) Convenient HSM connectivity for enstore, osm, tsm, preliminary for Hpss by BNL.

8 Admin & more Administration Miscellaneous Using standard 'ssh' protocol for administration interface. First version of graphical interface available for administration Large (optionaly) command set per module -> highly customizable. CRC checksum calculation and comparison (partially implemented yet) Pluggable door / mover pairs (gssdcap,gsiftp,http...) (can be extended due to clear interface definition) Data removed only if space is needed

9 LCG Storage Element DESY dcap lib incorporates with CERN s GFAL library gsiftp support SRM version ~ 1 (1.7) plus Space Reservation limited GRIS functionality (script based)

10 Resilient dcache no HSM backend (yet, but in preparation) Controls number of copies for each dcache dataset Makes sure n < copies < m Adjusts replica count on pool failures Adjusts replica count on scheduled pool maintenance Embedded farm node dcache (makes use of local disk space) In production at various places Will be introduced with next offical dcache.org

11 dcap Protocol and Implementation implements I/O and name space operations including 'readdir' works on mounted pnfs and URL like syntax available as standard shared object and preload library ls -l dcap://dcachedoor.desy.de/user/patrick positive tested for Linux, Solaris, Irix ( partially for XP) automatic reconnect on server door and pool failures supports read ahead buffering and deferred write supports ssl, kerberos and gsi security mechanisms Thread safe Interfaced by ROOT

12 dcache Functionality Layers GFAL Storage Element (LCG) Wide Area dcache Resilient Cache Basie Cache System Storage Resource Mgr Ftp Server (gsi, kerberos) Resilent Manager Gris dcap Client Pnfs (gsi,kerberos) dcap Server dcache Core Cell Package HSM Adapter

13 Insight dcache Insight dcache

14 dcache Module Layers Grid Layer Gris Storage Resource Mgr Access Layer dcap Client http / https (gsi,kerberos) dcap Server Ftp Server (gsi, kerberos) Extended central services Resilent Manager HSM Flush Manager Prestager Core Layer Pnfs dcache Core HSM Adapter Cell Package

15 dcache Core Module dcache Core -> PoolManager Finds best pool for each request. Cleaner Forwards 'rm'requests from pnfs filesystem to pools. PnfsManager Interface between dcache and filesystem. Door (Launcher) Starts appropriate door for incoming connections. Pool Does space management and launches 'movers'.

16 SRM dcap door Component Toolbox Pool dcap mover ftp mover gsiftp door Pool Manager Pool dcap mover ftp mover PnfsManager Pnfs Cleaner Pool dcap mover ftp mover

17 Classical Setup Head Node SRM http Engine dcap door admindoor gsiftp door Pool Manager PnfsManager Cleaner Pool Node(s) Pool Pool Node dcap mover ftp mover Pool Pool Node dcap mover ftp mover Pool dcap mover ftp mover Pool dcap mover ftp mover Pool dcap mover ftp mover Pool dcap mover ftp mover Pnfs

18 Grid Setup MISC Node Door / Pool Unit dcap door Pool SRM Prestager Resilient MGR dcap mover ftp mover Pool dcap mover dcap door Pool ftp mover dcap mover gsiftp door ftp mover Pool dcap door dcap mover Pool dcap mover gsiftp door ftp mover ftp mover Pool dcap doordcap mover Pool ftp mover dcap mover gsiftp door ftp mover Pool dcap mover ftp mover gsiftp door SRM Node dcap door gsiftp door Head Node http Engine Pool Manager Pnfs dcache, Storage Element and HSM optimizer admindoor PnfsManager Cleaner RAL, April 2005

19 Messages Door Pnfs Manager Pool Manager Pool HSM get metadata prepare dataset get DS from HSM copy DS to pool prepare transfer update cacheinfo Data Transfers

20 dcache The HSM Interface

21 Client Dataset life cycle dcache HSM precious Space needed cached File requested cached

22 Deferred HSM flush Precious data is separately collected per storage class Each 'storage class queue' has individual parameters, steering the HSM flush operation. - Maximum time, a file is allowed to be 'precious' per 'storage class'. - Maximum number of precious bytes per 'storage class' - Maximum number of precious files per 'storage class' Maximum number of simultaneous 'HSM flush ' operations can be configured Multiple HSMs instances and HSM classes are supported simultaneously

23 Deferred HSM flush (cont.) Tape Mount Data Transferred Data Flow Client -> dcache dcache -> HSM Time

24 The Pool Selection Mechanism Static Configuration Dynamic Behavior Tuning...

25 Pool Selection Mechanism Request Configuration DB Data I/O Direction Client IP number HSM (e.g. tape set) Decision Units Permitted Pools Subdirectory (tags) 0 1 Preferred Pool dcache vital functions Pool Space Costs Pool Load Costs

26 Write and reread... Expensive (safe) write pools Cheap commodity read pools

27 Pool Selection for Grid... DMZ Sub Cache External Subnet Grid Access Farm Subnet Central Cache

28 Selection for phy. sep. labs. Lab A Central Cache Lab B

29 Pool Selection : Tuning (1) Space vs. Load For each request, the central cost module generates two cost values for each pool : Space : Cost based on available space or LRU timestamp CPU : Cost based on the number of different movers (in,out,...) The final cost, which is used to determine the best pool, is a linear combination of Space and CPU cost. The coefficients need to be configured. Space coefficient << Cpu coefficient Pro : Movers are nicely distributed among pools. Con : Old files are removed rather than filling up empty pools. Space coefficient >> Cpu coefficient Pro : Empty pools are filled up before any old file is removed. Con : 'Clumping' of movers on pools with very old files or much space.

30 Pool Selection Tuning (2) Pool to pool transfers etc.... Panic Cost Level squeeze Not Recommended from tape Exited pool 2 pool Regular PANIC Take file from pool with lowest cost. Fetch file from Hsm to cheap pool, before delivering to it to client. Copy file to cheap pool first, before delivering it to client. low 0 Idle Take file from pool with lowest cost, otherwise try to get rid of duplicates.

31 Topic : checksum Supported Type : adler32 dcap calculated and sent to dcache (x) FTP can be sent by 'quote'' Calculated again checked and stored H S M Checked on READ Frequent CHECK on disk Calculated again and checked

32 coming soon... coming soon...

33 Coming soon... High speed mover queues Centralized HSM Flush ( disk spec aware) Scaling Web Monitors Smart Restore (Staging) nfs3 improved cacheinfo service Chimera (db based pnfs) SRM : Space reservation SRM : Version 2... Scalable StorageElement dcache using VOMS for authorization (authentication) Improved Documentation

34 Scalable Storage Element µ - Cache macro - Cache <= 10 TBytes <= 3 pool nodes ZERO Service no HSM > 150 TBytes > 80 pool nodes > 1000 pool nodes for resilient. full HSM support?

35 Improved dcache name service Stolen from Tigrans talk (chep interlaken) beta testing NFS 3 Production dcache Currently in Chimera NFS 2 Meta Data Database Pnfs Name Service

36 Improved HSM connectivity Improved (Pre)stager Collects requests in time bins Submits requests per 'tape' Knows about HSM load Central HSM flush monitor coordinates HSM mover streams Knows about HSM capability Known about disk capabilities

37 Data movement

38 dcache File Transfers H S M Authentication Module Namespace Provider (Pnfs) Door PoolManager Pool Mover Mover Application

39 Resilient dcache Replica Manager PoolManager Namespace Provider (Pnfs) Pool Mover Mover Pool Pool Pool Pool

40 dcache End of official presentation

41 Pool Selection Mechanism Pool Selection required for dcache dcache dcache dcache dcache Client HSM Client Pool selection is done in 2 steps I) Query configuration database : which pools are allowed for requested operation II) Query 'allowed pool' for their vital functions : find pool with lowest cost for requested operation

42 Pool Selection Mechanism Request Configuration DB Data Direction Client HSM dcache dcache dcache Client Client IP number Storage Class Directory Tags Pools sorted by Pref Mode A : fall-back only if all pools of pref. <x> are down. Mode B : fall-back if cost of pools of pref. <x> is too high.

43 Pool Group(s) Pool Selection Mechanism Storage Class Group(s) Directory Tag Groups Link Client HSM Client dcache Link preferences dcache dcache # <n> # <m> # <l> Subnet Groups

44 Pool Selection Mechanism Goals / Use cases Dedicated write pools (select by data direction) Allow 'precious' files on secure disks only. Read requests will trigger p2p to cheap disks. (e.g. datataking) Support multiple HSMs (select by storage class) Assign different pool set to different HSMs (e.g. HSM migration) Support 'group owned' pool sets (select by storage class or tag) Assign 'experiment data' to 'experiment owned pools' BUT have 'fallback' pools common to all experiments. Support 'working group' quotas (select by storage class or tag) Assign different number of pools to different working groups resp. 'data types' (raw,dst...) Special pools for farm subnets or external subnets e.g. : Grid users vers. Internal users.

45 Selection by subnet Central Cache DMZ Sub Cache External Subnet Grid Access Farm Subnet

46 HSM decoupling Dual Interface One high speed link per drive Mss Subnet Public Network H S M I H S M H S M II

47 Dynamic pool selection Method Frequent update of 'pools vital functions' - available space - least recently used 'timestamp' - number of movers (in,out,store,stage,p2p) Performing 'smart' guess between updates. Goal Uniform (even) distribution of requests per pool for requests coming 'in bunches'.

48 Pool Selection : Tuning (1) Space vs. Load For each request, the central cost module generates two cost values for each pool : Space : Cost based on available space or LRU timestamp CPU : Cost based on the number of different movers (in,out,...) The final cost, which is used to determine the best pool, is a linear combination of Space and CPU cost. The coefficients needs to be configured. Space coefficient << Cpu coefficient Pro : Movers are nicely distributed among pools. Con : Old files are removed rather than filling up empty pools. Space coefficient >> Cpu coefficient Pro : Empty pools are filled up before any old file is removed. Con : 'Clumping' of movers on pools with very old files or much space.

49 dcache End of official presentation

50 dcache Component License Model Storage Element Remotely accessible SRM Client Storage Resource Manager (SRM) Globus,Cog (GTPL) Resilient Cache Ftp Server (gsi, kerberos) COG (GTPL) Basie Cache System Resilent Manager dcap Client (gsi,kerberos) dcap Server COG (GTPL) BSD 3 rd party GPL? (library LGPL?)??? dcache Core Cell Package Sun Java VM (Sun Binary Code L) Pnfs Postgres Gdbm (BSD) (GPL)

51 DESY C Central Cache HH D2 Pre Cache HH Z Cache Zeuthen D2 D2 D2 D2 D2 D2 DMZ World C C C C C Local HH C C C C C D2 D2 D2 C C C C C C C C C C D2 Local Zeuthen Z Z Z

52 LCG Storage Element : File access Posix like I/O Replica Manager Client Physics Application Grid File Access Library (GFAL) SRM Client Local I/O dcap I/O rfio I/O Wide Area LRC SRM Service dcache SE dcap Service rfio Service Access GridFTP Service MSS Service Local Disk Source : Michael Ernst 18/5/2004