Data and Storage Services

Transcription

1 Data and Storage Services G. Cancio, D. Duellmann, J. Iven, M. Lamanna, A. Pace, A.J. Peters, R.Toebbicke CERN IT Department CH-1211 Genève 23 Switzerland

2 CERN IT Department CH-1211 Genève 23 Switzerland D.Duellmann from DM & SM TEG meeting 24.Feb.2012

3 CERN IT Department CH-1211 Genève 23 Switzerland D.Duellmann from DM & SM TEG meeting 24.Feb.2012

4 CERN IT Department CH-1211 Genève 23 Switzerland D.Duellmann from DM & SM TEG meeting 24.Feb.2012

5 CERN IT Department CH-1211 Genève 23 Switzerland

6 CERN IT Department CH-1211 Genève 23 Switzerland

7 Storage, Services, Data management Storage Services: object store, key value storage, cloud storage Includes (arbitrary) reliability layer Will not hide hardware performance, known latency, perfectly scalable Data management Data and file access Storage

8 Storage, Services, Data management Data and file access File access protocols, metadata management, can be mounted as a file system Secure, User authentication, Access control, Accounting, Quotas Interfaces to allow storage federation, redirection, data placement Data management Data and file access Storage

9 Storage, Services, Data management Data Management Global namespace, Hierarchical storage, automatic data placement, SRM, pinning Data management Data and file access Storage

10 Why multiple? Managed storage Castor Low latency, hundreds of metadata ops/sec, thousands of simultaneous use, high performance AFS Is this enough? Why more? 10

11 File / Storage levels of service? Cost Cost of hardware, energy, space Cost of operation team Reliability / availability Reliability: probability to lose data Availability: probability of not being able to temporarily access the data Performance Latency: Fixed delay per request Throughput: Transfer speeds seen by the client Scalability in term of size Scalability in terms of number of users Consistency: Different clients may see different data Richness of service: Auth, Autz, Quota, Accounting, Client side caching Namespace federation Performance Scalability Throughput 11 Cost Reliability Latency Consistency HW cost Ops Cost

12 Agile storage infrastructure Storage instantiated with custom levels of service: Large pools with low latency, multiusers, for analysis Highly available, highly reliable, multiuser pools, for condition data Low cost pools for scratch data that can be lost as already replicated (low reliability) Low cost pools, highly reliable, no multiuser, high latency, for archiving Service instantiated from (uniform) sets of unreliable hardware Resilient to hardware failures, human intervention limited to hardware resource provisioning, minimum cost of operation 12

13 Expected requirements in SRM, Hierarchical storage, pinning, automatic data placement On demand storage, negotiable level of Global access to files and data with metadata management, authentication, access control, accounting, quota Global access to scalable object store with no metadata management Interfaces to federate storage from independent clusters? 13

14 Which architecture? How much optimization is required? versus Data management Data and file access Storage Data and file access Data management Storage Openstack Huawei, EOS Hadoop AFS, EOS, Castor Castor

15 Which architecture? If we want to combine Storage and File systems versus versus Storage Data and file access Data and file access Storage Storage Data and file access S3 cloud service independent from AFS / EOS AFS / EOS delivered from S3 cloud service S3 cloud service as an additional interface on top of existing AFS / EOS Fully optimized Optimized for storage Optimized for file access

16 Facilities for storage Does storage require specific facilities or can it leverage from generic ones? On large facilities small systematic inefficiencies generate important costs Many of the existing infrastructures provide classes of hardware to match service requirements (test needed, storage cannot move, relies on replicas Understand gains versus increased service complexity) Storage Batch Database Facilities management (common hardware)

17 Questions to answer: What is the optimal architecture to deliver these in the next years? What are the tests, and the research activities we are need to do to answer the challenge above? How do we get there?