Using S3 cloud storage with ROOT and CernVMFS Maria Arsuaga-Rios Seppo Heikkila Dirk Duellmann Rene Meusel Jakob Blomer Ben Couturier
INDEX Huawei cloud storages at CERN Old vs. new Huawei UDS comparative Multi-client ROOT benchmarks Real CernVMFS application Summary 14/04/2015 Maria Arsuaga-Rios CERN openlab 2
NEW HUAWEI UDS GENERATION Old generation 384 disks 768 TB 7 front-end nodes S3 multipart uploads not supported New generation 300 disks 1.2 PB 4 front-end nodes Improve S3 support: multipart uploads Less rack space More compact storage nodes 14/04/2015 Maria Arsuaga-Rios CERN openlab 3
Throughput tests 100MB downloads 100MB uploads Metadata tests 4kB downloads 4kB uploads OLD vs. NEW HUAWEI UDS COMPARATIVE 14/04/2015 Maria Arsuaga-Rios CERN openlab 4
OLD vs. NEW HUAWEI UDS Throughput Download Tests Download speed [GB/s] 3 2.5 2 1.5 1 0.5 1 front-end 2 front-ends 3 front-ends 4 front-ends 5 front-ends 6 front-ends 7 front-ends Full network bandwidth = 20Gb/s Front-ends scale up Old UDS Overstressed 0 0 20 40 60 80 100 120 140 160 180 200 220 Concurrent clients [count] 14/04/2015 Maria Arsuaga-Rios CERN openlab 5
Download speed [GB/s] 3 2.5 2 1.5 1 0.5 1 front-end 2 front-ends 3 front-ends 4 front-ends OLD vs. NEW HUAWEI UDS Throughput Download Tests New UDS Full network bandwidth = 20Gb/s Front-ends scale up 0 0 20 40 60 80 100 120 140 160 180 200 220 Concurrent clients [count] 14/04/2015 Maria Arsuaga-Rios CERN openlab 6
OLD vs. NEW HUAWEI UDS Metadata Download Tests Download speed [1,000 files/s] 30 1 front-end 2 front-ends 25 20 15 10 5 3 front-ends 4 front-ends 5 front-ends 6 front-ends 7 front-ends Old UDS Front-ends scale up 0 0 200 400 600 800 1000 1200 1400 1600 Concurrent clients [count] 14/04/2015 Maria Arsuaga-Rios CERN openlab 7
Download speed [1,000 files/s] 30 25 20 15 10 5 1 front-end 2 front-ends 3 front-ends 4 front-ends OLD vs. NEW HUAWEI UDS Metadata Download Tests New UDS Front-ends scale up 0 0 200 400 600 800 1000 1200 1400 1600 Concurrent clients [count] 14/04/2015 Maria Arsuaga-Rios CERN openlab 8
MULTI-CLIENT ROOT BENCHMARKS Full parametric Heterogeneous clients Access protocol compatibility Multi-client ROOT benchmarks Offline analysis Concurrent client accesses to real ATLAS ROOT files Sparseness, entries accessed 100%, 0.1%, 0.001% 2 use cases: 1 bucket 64 buckets Tested only in UDSs Ceph and Amazon do not support multi-range requests 14/04/2015 Maria Arsuaga-Rios CERN openlab 9
MULTI-CLIENT ROOT BENCHMARKS % Entries Total Entries = 11918 Branches = 5860 Size = 760 MB Total Entries Read Calls MB read NTuple ATLAS W and Z bosons Analysis 100% 11918 23 758.42 0.1% 12 12 320.92 0.001% 1 5 42.08 14/04/2015 Maria Arsuaga-Rios CERN openlab 10
MULTI-CLIENT ROOT BENCHMARKS 1 Bucket Old UDS New UDS Full network bandwidth =20Gb/s Does not scale as expected Scales up until too many clients access the same small amount of data! 14/04/2015 Maria Arsuaga-Rios CERN openlab 11
MULTI-CLIENT ROOT BENCHMARKS 64 Buckets Old UDS New UDS Full network bandwidth =20Gb/s Full network bandwidth =20Gb/s 14/04/2015 Maria Arsuaga-Rios CERN openlab 12
MULTI-CLIENT ROOT BENCHMARKS Summary Full bandwidth is reached by executing sparse accesses with both UDS generations UDS supports multi-range get requests which are common in HEP analysis E.g. Ceph and Amazon S3 do not support multi-range requests 14/04/2015 Maria Arsuaga-Rios CERN openlab 13
REAL CernVMFS APPLICATION Introduction What is CernVMFS Read-only cached file system to deliver software Widely used in WLCG (Worldwide LHC Computing Grid) Mounted by users and files are downloaded on demand CernVMFS back-end challenges Publishing new software should be as fast as possible Files are accessed through the HTTP 14/04/2015 Maria Arsuaga-Rios CERN openlab 14
REAL CernVMFS APPLICATION S3 Backend New features to CernVMFS (release 2.1.20) S3 compatible storage Supports multiple buckets and accounts LHCb nightly builds test case LHCb software (binary and source files) Releases done daily for over 3 months ~1 million files in release (~150k new files) Completed successfully with UDSs and Ceph 14/04/2015 Maria Arsuaga-Rios CERN openlab 15
REAL CernVMFS APPLICATION Deletion Process Deletions should not affect the cloud storage functionality Old UDS does not handle deletions correctly New UDS is fast enough to keep up with average LHCb nightly builds repository size growth Old versions can be deleted to make space for new ones 14/04/2015 Maria Arsuaga-Rios CERN openlab 16
SUMMARY Scalable throughput and metadata performance verified Successful scalability with different ROOT access patterns LHCb nightly builds software stack distributed with CernVMFS using UDS backend 14/04/2015 Maria Arsuaga-Rios CERN openlab 17
THANK YOU 14/04/2015 Maria Arsuaga-Rios CERN openlab 18