From Research To Transfer: User-Controllable Cloud Storage

Transcription

1 Department of Computer Science Institute of Systems Architecture Chair of Computer Networks From Research To Transfer: User-Controllable Cloud Storage Josef Spillner 14th TERENA TF-Storage Meeting February 10-11, 2014, Zurich, Switzerland

2 The Trouble with Cloud Storage proprietary client backup/sync vs. storage quality? trust? institutional utility billing web access only pricing: progressive/ degressive file size limit 3 months limit; tariff choice

3 Taming the Trouble: RAICs, RAOCs proxy cloud controller/ gateway storage targets: services, devices service properties service bundles integration scheduling optimisation auto-selection auto-configuration auto-repair extension backup sync sharing...

4 The Cloud Storage TU Dresden Short history: research prototypical software development 2010 ideas and service modelling 2011 NubiSave storage controller started UCC paper on Cloud storage controllers 2012 Usable software, Debian packaging 2013 NubiVis; π-box personal Cloud VM FGCS article on optimal storage SUCRE summer school tutorial at KIT 2014 NubiGate VM experiments and pilot installations transfer widespread production deployments

5 Research

6 Storage Service Ecosystem Excursus: Service Science -> Bundles -> RAOC functionality super-domain Infrastructure domain 2 domain 1 domain Cloud Storage non-functional properties nfp 1 sub-domain File Storage nfp 2 consumer client SLAs protocol description service interface messages service instance service impl engineer service hosting provider nfp backup

7 Storage Service Descriptions Base Ontology (WSML) concept MeasurementUnit conversionfactor impliestype _double concept SpaceUnit subconceptof MeasurementUnit instance GB memberof SpaceUnit conversionfactor hasvalue Instance Ontology webservice GoogleStorage importsontology { _"urn:ontology:conqo/cloudqos.wsml#" } capability ServiceCapability postcondition definedby?servicetype memberof cloud#cloudstorage. instance PricePerData memberof { cloud#priceperdata, qos#servicespec } qos#value hasvalue 0.17 qos#unit hasvalue qos#euro

8 Storage Service Bundles $$ a Service bundle x replication (k+m) erasure coding s1 s2 s3 s4 s5 s1 secret sharing s6 s1 s2 s3 s4 s5 s6

9 Storage Service Markets

10 Storage Flows sink transport source Storage Targets Data Flow Integration USB stick local directory NFS export WebDAV Commercial (Dropbox) block modifier (compress) Splitter/ Multiplexer File modifier (enc,dedup) local directory block modifier (stego) CIFS proxy Splitter/ Multiplexer Splitter/ Multiplexer file modifier (enc) Splitter/ Multiplexer web application

11 Reconfigurable Storage Flows... alias Software-Defined Storage Modules (i.e. FUSE modules): - Splitter (1:n) - Modifiers (1:1) * encryption * compression *... - Transports * Directory * CIFS, NFS * WebDAV * SSHfs * S3fs *... * CloudFusion - SDS propagation

12 Multi-User Storage Gateways FlexDDPL: Flexible Data Distribution Policy Language scopes applied in contexts (e.g. user database) ~user!negation T:22:00 fragment mime:message/rfc822 rules applied to targets (e.g. storage provider) store control adapt

13 Feature: Algorithms Selection Background secret sharing forward error correction / replication Striping ( RAID0) capacity 1.0, safety 1.0, performance 2.0 Mirroring ( RAID1) capacity 0.5, safety 2.0, performance 1.0 XOR parity ( RAID5, RAID6) 4 disks: capacity 0.75, safety 1.33, performance 1.0 Erasure coding ( RAIDn) n storage targets, k significant, m redundant: capacity k/m, safety 1+m/k (Rotated) Reed-Solomon: maximum-[hamming-]distance separable (MDS) property Cauchy-Reed-Solomon, Vandermonde-Reed-Solomon: practically secure; CRS faster due to XOR operations compared to GF(2w) AONT: information-theoretically secure AONT-RS: blend of AONT with (C)RS; alternative: encrypted fragments

14 Feature: Nested Contexts Configuration Data context»normal«context»private«

15 Feature: Weights Assigning weights to the storage targets to fully utilise the capacity to exploit faster upload connections

16 Feature: Recursion Saving the database which contains all file and fragment metadata... to avoid single point of failure to allow for selective sharing

17 Feature: Streaming (1:x) Continuous operation during read or write of large file buffers on controller device can be kept small improved performance through parallel coding and transmission

18 Feature: Batching (x:1) Encoding speed of different batch sizes 15 nodes Encoding [MBs] M=0 M=4 M=8 M= Filechunk size [KB]

19 Experiments Nubisave Read Performance - 1MB File - UseAllInParallel Read in KB/s Redundancy 70 peacs: Performance and Efficiency Analsis for Cloud Storage Experiment Week Feb Storages 2 1

20 Autonomous Storage Clients Scenario: A robot wants to store some data... Labcast available: watch?v=ndln0fg9okk

21 Autonomous Storage Clients Scenario: Pervasive storage on all devices Credentials may be auto generated

22 Software

23 Comparison of Storage Controllers Prototype Origin Authors Approach Integration Public Availability Dependable Sky ('11) T-Clouds, Univ. of Lisbon Bessani, Correia, Quaresma, André, Sousa data units, byzantine fail, Rabin erasure application library no ('11) HPI, Potsdam Maxim Schnjakin, Christoph Meinel NFPs, propr. platform, Liberation erasure web interface no TrustedSafe ('10) egovcd, Berlin/ FhG FOKUS (no publication) n fragments: one on USB, others remote Windows plugin no Cloud Shredder ('11) Chinese Academy of Sciences Nan Zang, Jiwu Jing, Peng Liu Two fragments: one local, one remote (not yet known) no SecCSIE ('11) T-Systems MMS, Dresden Ronny Seiger, Stephan Groß Cachy ReedSolomon erasure CIFS proxy no RACS ('10) Cornell University Abu-Libdeh, Princehouse, Weatherspoon erasure coding: Zfec ReedSolomon S3 proxy, can be distributed w/ Zookeeper yes ( since 2012, community projects: plus numerous commercial implementations: NetApp StorageGRID, Cleversafe, Trusted Safe etc. Tahoe-LAFS externals, TCD, OpenStack/Intel - PyEC2Lib

24 OpenStack Intermezzo

25 NubiSave Cloud Storage Controller Installation of NubiSave Git repository Debian packages git://nubisave.org/git/nubisave Integrated into π-box or NubiGate VMs

26 NubiSave Storage Flow Editor

27 NubiSave Configuration

28 NubiSave Configuration

29 NubiSave Tools Controller operation Storages and database $ nubisave [<instance>] $ nubisave-status Master script which starts both of the below combined. Lists all splitter instances and storage targets attached to them or as part of any storage flow. $ nubisave headless Starts a new instance of the splitter/dispersion file system. $ nubisave gui Starts the storage flow editor with storage integration configuration. $ nubisave-mounter [<module>] Mounts all splitters and/or storage targets. $ nubisave-unmounter [<module>] Unmounts; inverse of the above. $ nubisave stop $ nubisave-database [...] Stops the splitter. Inspection of storage metadata.

30 Splitter-NG Framework Plugins: Jerasure, JSharing, RAID-1, more coming...

31 Transport Module: CloudFusion

32 Additional Implementations

33 NubiVis: Distributed Data Visualisation

34 NubiVis Map View

35 NubiGate VM: Everything in one Box

36 NubiGate VM: OwnCloud Frontend Our instance RN-Dropbox /owncloud Your instance...?

37 Future Vision

38 Future Work What's next in our lab Software improvements Configuration through the web interface (student thesis) Git-based service registry query Research: MapReduce-like processing over dispersed data Scaling large with more users and storage providers, HA setups etc. Federation between research institutions: reciprocative/collaborative storage resource sharing <-- role of NRENs

39 Publications [SS13] Josef Spillner, Alexander Schill: Orchestration of Distributed Storage Targets through Storage Flows. 5th IEEE International Conference on Cloud Computing Technology and Science (CloudCom), Bristol, UK, December [SQ+13] Josef Spillner, Maximilian Quellmalz, Martin Friedrich, Alexander Schill: peacs - Performance and Efficiency Analysis for Cloud Storage. Workshop on Cloud Storage Optimisation 2nd ESOCC, Málaga, Spain, September [SMS13] Josef Spillner, Johannes Müller, Alexander Schill: Creating Optimal Cloud Storage Systems. Elsevier Future Generation Computer Systems (FGCS), Issue 29(4), p , June DOI /j.future [SS12] Josef Spillner, Alexander Schill: Flexible Data Distribution Policy Language and Gateway Architecture. 1st IEEE Latin American Conference on Cloud Computing and Communications (LatinCloud), Porto Alegre, Rio Grande do Sul, Brazil, November [SP+12] Josef Spillner, Christian Piechnick, Claas Wilke, Uwe Aßmann, Alexander Schill: Autonomous Participation in Cloud Services. 2nd International Workshop on Intelligent Techniques and Architectures for Autonomic Clouds 5th IEEE/ACM UCC, Chicago, Illinois, USA, November [SG+11] Josef Spillner, Gerd Bombach, Steffen Matthischke, Johannes Müller, Rico Tzschichholz, Alexander Schill: Information Dispersion over Redundant Arrays of Optimal Cloud Storage for Desktop Users. 4th IEEE/ACM International Conference on Utility and Cloud Computing (UCC), Melbourne, Australia, December [Best Paper Award]