How To Write A Dcache File To A Flash Memory On A Flash Disk (Dcache)

Transcription

1 dcache, large scale-out affordable storage On behave of the team 1

2 Content People and Funding Deployment Supported protocols Data access Storage control Managed storage Design facts 2

3 People Funding 3

4 People and Funding HGF&DGI 3 EMI 2 DESY NDGF, Copenhagen 1 DESY 2 1 SNIC, Linköping Chicago FERMlab 2 En Plus HGF : Helmholtz Alliance DGI : German e-science EMI : European Middleware Initiative Centralized dcache support for Germany : Government funding Aachen Munich Wuppertal Karlsruhe Berlin 4

5 Two words on EMI European Middleware Initiative 5

6 Emi Factsheet EMI Factsheet Budget : about 24 Million Euros Funding : about 50% by EU- FP7, rest by partners Covers : JRA, SA and NA Partners : 22 Middlewares: Arc, glite, UNICORE and dcache 16/09/2010 EMI Overview - EGI TF, Amsterdam 6 May 25, 2011 EMI Data, IEEE and MSST, Denver 6

7 Deployment WLCG Others 7

8 WLCG : World Wide LHC computing Grid LHC : Large Hadron Collider WLCG 8

9 WLCG : Specification Trying to find out why we are heavy (Higgs) Maybe more about dark matter and dark energy Producing about 15 PBytes per year Raw data is stored at CERN and at least at another Site (Tier I) Legend Tier 0 : CERN Tier 1 : CounZes (11) Tier 2 : about 200 9

10 Status : dcache deployment o o o 94 PB in total 7 Tier I s 40 Tier II s Dresden Munich Freiburg Wuppertal Mainz Göttingen Aachen DESY KIT Germany 16 PB Barcelona Lyon Athens Europe 44 PB Pisa Roma NDGF FERMIlab Florida BNL USA 28 PB Madrid Amsterdam Séminaire Aristote : Big Data, Ecole Polytechnique, Patrick Fuhrmann WLCG Storage per SE type dcache Purdue Madison Wisconsin Cambridge, MA East : 1 PB Sweden London Other Storage Systems 10

11 Largest dcache Installation (BNL) Brookhaven National Lab (New York) Information provided by Hironori Ito (BNL) 80 Million files in total 85 storage hosts with about 600 pools. (dcache storage unit) Total space on disk : 8.8 PBytes (Used are 7 PBytes) Total space on tape (HPSS, IBM) : 2.5 PBytes File create 1000 / min = 17 / sec Storage Network Two weeks One year (10/11) 11

12 Most interesting deployment The 7 biggest Nordic Computer centers form a single Tier I Many different tape back systems in different countries. Resources are scattered (CPU & Storage) Services can be centralized Advantages in redundancy Especially in 7*24 hour data taking Slide stolen from Mattias Wadenstein, NDGF 12

13 Other Deployments Historically DESY : HERA (Zeus, H1) FERMILAB : Tevatron (CDF) Sloan Digital Sky Survey 13

14 New data intensive communities Low Frequency Array Is using dcache at SARA (Amsterdam) and Jülich (Germany) Center for Free Electron Laser Science Would like to use dcache at the DESY storage. Swedish National Infrastructure for Computing Using dcache for Swedish academic purposes. 14

15 Supported Protocols 15

16 New Protocol support dcache http(s) / WebDAV NFS 4.1 / pnfs Cloud Data Management Interface xroot/dcap protocol Global Namespace Séminaire Aristote : Big Data, Ecole Polytechnique, Patrick Fuhrmann Not yet decided Standards : Useful For Many communities WLCG Only 16 Not yet decided

17 Protocol support : WebDAV Very useful for new (non- LHC) communizes. ITEF Standard Allows File system like access with Mac OS Linux Windows 17

18 NFS v 4.1 / pnfs My favored Topic 18

19 Protocol support : NFSv4.1/pNFS CITI, at the University of Michigan, is funded by major storage providers to coordinate the pnfs effort and provide reference implementazons. Group meets three Zmes a year to check interoperability. 19

20 Protocol support : NFSv4.1/pNFS Stolen from : hgp:// 20

21 Protocol support : NFSv4.1/pNFS Stolen from : hgp:// Benefits of Parallel I/O Delivers Very High Application Performance Allows for Massive Scalability without diminished performance Benefits of NFS (or most any standard) Ensures Interoperability among vendor solutions Allows Choice of best-of-breed products Eliminates Risks of deploying proprietary technology 21

22 Protocol support : NFSv4.1/pNFS Simplicity Regular mount-point and real POSIX I/O Can be used by unmodified applications (e.g. Mathematica..) Data client provided by the OS vendor Smart caching (block caching) development done by OS vendors Performance pnfs : parallel NFS (first version of NFS which support multiple data servers) Clever protocols, e.g. Compound Requests 22

23 dcache is Managed Storage Manual Storage management o o The Storage Resource Manager (SRM) Storage Migration Module Automatic Storage Management o o o Storage Attribute by directory Hot Spot detection Resilient Manager 23

24 dcache Idea Clients NFS 4.1 WebDAV gridftp Pool Pool Pool dcache Head node Pool Pool Pool Tape Interface Tape Interface Tape Libraries 24

25 Normal dcache file cycle (Storing data) Write file to dcache, using any supported protocol. Files on disk are precious Pool collects data and flushes to tape, following rules. Files on disk become cached cached files are automatically removed if space is running short. Pool Pool Pool Pool 25

26 Normal dcache file cycle (retrieving data) Pool Pool Pool In case of a cache miss : The file is automatically restored from tape and subsequently delivered to clients, using the selected protocol. 26

27 Managed Storage Final destination determined by directory myspace/ mytape mydisk Disk Tape /x/myspace/mytape/foo /x/myspace/mydisk/foo 27

28 Managed Storage : SRM The storage resource manager protocol SRM 2.2 : defined by the Open Grid Forum (OGF) Defines storage media (Disk/Tape) Can use Spaces (similar to AMAZON buckets) with attributes (disk, tape, size) Pin / Unpin files Bring Online file(s), in preparation for a read. Remote secure protocol with many implementations 28

29 More Managed Storage Automatic file replication on hot spot detection If a pool is used heavily, dcache starts to spread files from this pool to other (lazy) pools. Resilient manager On basis of a pool set, a minimum and maximum number of replicas for all files can be defined. dcache automatically adjusts the replicas if pools go down or are scheduled for maintenance. Migration Module Files and be shuffled around between pools (by rules) to allow to spread load or decommission pools. 29

30 dcache allows to make use of different storage capabilities Access Latency dcache pools To Clients Tape Tape like Hadoop FS S3. Regular DDn, Dell SSD s 30

31 Design Details 31

32 dcache, in layers Standard File Access Protocols http(s) WebDav DISK NFS 4.1 Common Security Layer DISK gsiftp Authentication : Kerberos, X509, Password Authorization : ACL s for File system and storage control (SRM) multi-media storage layer SSD SSD Storage Management SRM Unified ID management Common Name Service Layer Extended Names Service Queries (SQL) Tape 32

33 Authentication X509 Certificates Proxies FQAN (Group/ Role) SRM Perhaps Translator User <password> Kerberos

34 No Secrets Anymore All Java Name space abstraction Legacy implementation (PNFS) or New Implementation (any JDBC DB, def. postgres) Component communication via message passing: Private Protocol (Cells) or Java Messaging Service (JMS) Scalable components : Protocol Endpoints and data pools Single Point of failures : namespace and pool/space manager With (2 nd Golden Release) : Very nice configuration system 34

35 Conclusion dcache is about storing, accessing and managing huge amounts of data. Depending on the configuration (resilient manager) you may use cheap hardware. Historically tuned for HEP and WLCG For about 2 year focusing on more communities, which have been committing themselves to standards (web 2.0) dcache collaboration nicely distributed amongst Europe and the US. Funding spread amongst different bodies. (e.g EMI) More contributions/contributors welcome. 35

36 Further Reading www. 36