XtreemStore A SCALABLE STORAGE MANAGEMENT SOFTWARE WITHOUT LIMITS YOUR DATA. YOUR CONTROL

Transcription

1 XtreemStore A SCALABLE STORAGE MANAGEMENT SOFTWARE WITHOUT LIMITS YOUR DATA. YOUR CONTROL

2 Archive Manager - the Basis for XtreemStore DMS / Files ScienDfic Others PACS VIDEO PrePress CAD/CAM NFS / CIFS TCP/IP (remote) Archive Server Incl. Performance Disk SAN (remote) Disk archive Disk Media Disk archive Disk Media Tape archive Tape Media Tape archive Tape Media

3 Several Disk and Tape Pools UNIX WINDOWS LAN Archive Manager Remote Copy Disk Media Tape Media Tape Media ParDDoning for physical separadon of data

4 Key Features Online access to unlimited content MulDple media strategy - Secure and cost effecjve archiving on tape and / or removable disk - fast access to disk ParDDoning for different applicadons ( data separadon ) Remote copy funcdonality Several levels of backup / recovery opdons Free selecdon of archival hardware (server, disk, tape libraries) Direct access through file system (addidonal API opdonal) NaDve support for CIFS (AcDve Directory) & NFS

5 Need for high Performance POSIX Archive POSIX interface is very common to all applicadons But Standard File System have limitadons ArchiveManager inherits those limitadons when working with Standard FIlesystems Major limits: - The amount of files within one File System - The throughput of one single File System - Throughput for huge files (10TB and more) To digest large data volumes from Lustre, GAM needs an embeded adapted parallel file system (P/HSM FS) - Everything needs to be parallel GAM + P/HSM FS à XtreemStore

6 New Product: XtreemStore ArchiveManager (GAM) with built in Parallel HSM File System à XtreemStore Storage Management So^ware (HSM; Archive; Backup) for high performance applicadons such as - Lustre - BeeGFS - Others Parallel access through a Meta File System with POSIX interface to unlimited amount of storage Grid structure build on standard PC hardware

7 So^ware Architecture XtreemStore NFS / CIFS NFS / CIFS NFS / CIFS XtreemStore Parallel HSM File System GAM Client GAM Server Driving an unlimited amount of standard hardware

8 How it works High Performance ApplicaDon High Performance ApplicaDon High Performance ApplicaDon XtreemStore i.e 10 TB file à Unlimited Global namespace (One logical file system) HSM funcdonality for Disk and Tape

9 Exchange Lustre - XtreemStore Lustre offers access to a file with a plurality of streams XtreemStore offers access to a file with a plurality of streams To take advantage of this parallel access opdons for data transfer between such systems, a copy tool is required that takes use of it XS- Parallel is designed and implemented therefore

10 Parallel POSIX Source Node Target Node Source Node Target Node XS - Parallel is just so^ware Storage adaptors allow to connect different types of storage Actually there is only POSIX implemented The so^ware may run on - the source machine - the target machine - on dedicated computer nodes The amount of streams running in parallel is not limited

11 HA through Distributed Control InformaDon Master CI CI CI CI CI CI A Master node supervises all nodes. Control informadon is copied to every node Each node may take control when master fails Running jobs are finished, even if several nodes fail

12 HSM for Lustre with API Overview Lustre Clients API Lustre Server Parallel Master XtreemStore P/HSM FS GAM Clients GAM Server Primary Storage Archive Storage Disk / Tape

13 FuncDonality XtreemStore offers a transparent integradon into Lustre That means data moved into the archive is sdll transparently accessible through Lustre MigraDon of old data from Lustre into to the archive is managed by predefined policies Recall (unmigrate) is driven by a transparent access to a file File transfer from Lustre to Archive and back may be executed in parallel through unlimited amount of nodes

14 Throughput A node based on standard hardware is expected to run a sustained data rate above 350 MB/second XtreemStore scales almost linear with the amount of nodes So the performance target for a system with 20 storage nodes is expected with: - HSM - throughput per hour = 22 TB - HSM - throughput per day = 500 TB A system with 40 nodes should be able to run as much as à 1 PetaByte per day

15 Hardware Structure Disk & Tape from any Vendor Lustre ApplicaDon Storage node 1 Storage node 2 Storage node n JBOD or Cheap disk Tape Library

16 XtreemStore Product Family = Storage Management for Lustre Lustre Clients & Server Robin Hood Policy Engine HSM API Symbolic Link Lustre Server Parallel Storage Adapter Backend Technologies Object Store S3, etc. NAS Isilon etc. p/hsm FS ArchiveManager (GAM) Disk & Tape From any Vendor XtreemStore

17 Status of XtreemStore Products CooperaDon with Intel (earlier Whamcloud), CEA and GRAU DATA started in May 2012 XtreemStore Parallel for Isilon - planned GA Oct XtreemStore HSM planned GA Dez. 2014

18 YOUR DATA. YOUR CONTROL