National Data Storage data replication in the network
|
|
- Domenic Franklin
- 8 years ago
- Views:
Transcription
1 National Data Storage data replication in the network Maciej Brzeźniak, Michał Jankowski, Norbert Meyer, PSNC, Supercomputing Dept. 1st Technical meeting in Munich, December 5-6th, 2011 Project funded by: NCBiR for under KMD2 project (no. NR /2011) Full Polish name of the project: System bezpiecznego przechowywania i współdzielenia danych oraz składowania kopii zapasowych i archiwalnych w Krajowym Magazynie Danych Project partners 10 Polish universities and supercomputing centres:
2 National Data Storage NDS Overview: NDS Architecture: Design assumptions Overal architecture Data replication in NDS Data Replication modes Replication protocols usage User profiles vs data replication settings Rule-based replication? NDS vs external world vs EUDAT NDS future: NDS2 secure data storage and exchange
3 NDS - design assumptions Overall assumptions: Avoid SoF - distributed: data & meta-data replication Standard access protocols(tools) to be usable Abstraction of system internals Logical namespace visible to user; Separate namespaces for different user groups Robust implementation(c/c++) within 2 years Tape systems (HSMs) on the back-end (for cost-effieciency) Main applications Archival and backup data storage Effective storage and accesss of large files Multiple small files not welcome
4 NDS Project status National Data Storage (R&D project: ) System architecture & concept Software stack (rpms for CentOS/RHEL) Current NDS deployment: Backup and Archive Services for Science BADSS (Service Platform for e-science) Capacity: 12,5 PB of tapes in 5 sites & performance: 2 PB of disks in 5 sites National Data Storage 2 (R&D project: ) Secure storage and data sharing (user-side encryption + integrity control)
5 NDS highlights Automated, TRANSPARENT, data replication Users do not see the details (if they don t want; they can) They speak to remote virtual filesystem Abstract data access interfaces: File-system view of the data (remote virtual filesystem) NDS is implemented as a user-level code (FUSE library) User access: standard methods: SFTP, WebDAV, GridFTP Storage access: NFS / GridFTP-NFS (each SN exposes at least NFS) Meta-data replication: Automated, transparent Postgress Slony-I + semi-synchronous replication DR, not full HA (no recovery automation)
6 NDS architecture (1) Overall picture User Metadata DB Database Access Methods Servers (SSH, HTTPs, WebDAV...) VFS for data and meta-data NDS system logic Access Users DB Accounting & limits DB Replica access methods servers (NFS, GridFTP) FS with data migration (HSM) Replication Storage Storage HSM system (NFS) NAS appliance
7 NDS architecture (2) Data replication & presentation User Metadata DB Database Access Methods Servers (SSH, HTTPs, WebDAV...) VFS for data and meta-data NDS system logic Access Users DB Accounting & limits DB Replica access methods servers (NFS, GridFTP) FS with data migration (HSM) Replication Storage Storage HSM system (NFS) NAS appliance
8 NDS architecture (3) Data replication & presentation Data Daemon Implements the core NDS system logic (together with MC): I/O serving, filesystem presentation data operations with replication meta-data-related operations Emulates Virtual File System Supports most of POSIX functions: open, close, read, write opendir, readdir, getattr, setattr, rename, link, unlink... Based on FUSE (Filesystem in USErspace) Additional: enforces security policies (access control) optimizes replica access and creation implements limits and accounting
9 NDS architecture (4) Async. vs sync. replication from VFS perspective: Writing to the system (async mode) VFS: OPEN (new file, O_RDWR O_CREAT) - Register a new logical file in MC (lock for writing) - Create one physical replica - Register replica in MC VFS: WRITE... - Write to localreplica(async.) (QUICK-local, single replica write) - Update meta-data (size, last access etc.) VFS: CLOSE (on anopenedfile) - Flush buffers and close replica(async.) (QUICK) - Update meta-data incl. release write locks - Return to user Asynchronous action: Make replicas - Enqueue replication tasks to replication daemon - Update meta-data - Replications daemon (in the background) does the replicas (typically 3-rd party: SN1->SN2) Writing to the system (sync mode) VFS: OPEN (new file, O_RDWR O_CREAT) - Register a new logical file in MC (lock for writing) - Create physical multiple replicas - Register replicas in MC VFS: WRITE... - Write to all replicas(sync.) (TAKES TIME-remote, multiple sites to write) - Update meta-data (size, last access etc.) VFS: CLOSE (on anopenedfile) - Flush buffers(also to remote replicas) (TAKES TIME) - Update meta-data incl. release write locks - Return to user All replicas already done
10 NDS architecture (5) Replica access methods (AN-SN) (1) Access Methods Servers (SSH, HTTPs, WebDAV...) VFS for data and meta-data NDS system logic Replica access client (NFS) Replica access client (GridFTP) Access Low latency High bandwidth (eg. 10 Geth) LAN WAN High latency High bandwidth (eg. 1 Geth) NFS GridFTP NFS GridFTP or NFS used when needed LOCAL Storage Replica access method (GridFTP) Replica access method (NFS) Replica access method (GridFTP) Replica access method (NFS) REMOTE Storage
11 NDS architecture (6) Replica access methods (AN-SN) (2) NFS and GridFTP used where they fit best: Static protocol selection (currently), Dynamic protocol selection e.g. basing on file size (planned) NFS State-less, IOPS friendly Lowoverheadon IOPS operations: small files access meta-data related operations Low performance(mb/s) on long-distance No parallelism (to/from single file) NFS 4.1. (pnfs) on the horizonbut stillnot there Usage in NDS: meta-data related operations accessing replicas on local SNs access to small files on remote SNs (future) Stable, standardised GridFTP State-full, can exploit available badwidth High overhead on IOPS operations: even small files access and meta-data ops require session High performance (MB/s) despite distance Parallelism (up to 256 streams) 64+ streamscansustain1geth link (1000 km-long) Usage in NDS: 3-rd party replication(async. mode) transferring replicas to/from remote SNs Stability issues, even if standard in Grids
12 NDS architecture (7) GridFTP 3-rd party replication (SN-SN) (3) 3-rd party transmission (SN->SN) used in async. replication mode Access Methods Servers (SSH, HTTPs, WebDAV...) VFS for data and meta-data NDS system logic Replication daemon (GridFTPclient) Access Low latency High bandwidth (eg. 10 Geth) LAN WAN High latency High bandwidth (eg. 1 Geth) GridFTP control conection LOCAL Storage Replica access method (GridFTP) WAN High latency High bandwidth (eg. 1 Geth) Replica access method (GridFTP) REMOTE Storage
13 NDS architecture in PLATON: Replica access: GridFTP-NFS access to SNs Access Methods Servers (SSH, HTTPs, WebDAV...) VFS for data and meta-data NDS system logic Replica access client (GridFTP) Replica access client (NFS) Access GridFTP GridFTP virtual Storage Replica access method (GridFTP) NFS client Replica access method (GridFTP) NFS client virtual Storage NFS NFS HSM system (NFS) NAS appliance(nfs) FS with data migration (HSM)
14 NDS architecture (8) Replication-related settings (1) Replication-related parameters of profile: Profile parameter Possible values and meaning Replication mode Asynchronous (default) Synchronous Number of replicas Typically 2 (max. 3) Can be set to any value Allowed storage sites and nodes: Default replica locations Additional replica location Storage media type: Disk vs tape (HSM) Replicas are typically created in default locations Additional replica locations are used in case of failure of default ones Using a combination of allowed storage sites & nodes + knowledge on the deployment infrastructure we can determine media type
15 NDS architecture (9) Replication-related settings (2) Replication is configured per-profile, NOT per data-object, e.g. directory / file Policies are static cannot be changed dynamically Users can use one or many profiles: Assigned to profiles using DNs of certs => multiple certificates have to be used in order to access different profiles Fast, HA & FT spacefor backups: Replication: SYNC 3 replicas on (1 local + 2 distant) on disks only FT space for archives: Replication: ASYNC 2 replicas on distant nodes; both copies on tapes Safestoragespacefor collaboration: Replication: ASYNC 2 replicas: local on disk + 1 remote in HSM
16 NDS features vs EUDAT (1) Automated, TRANSPARENT, data replication Safe, transparent replication service case service Abstract data interfaces above and below NDS: SFTP, WebDAV, GridFTP for users Possibleto interfacewith NDS from other systems e.g. 3-rd party transfer to/from NDS Data available through VFS layer on ANs: Possible to add new access methods Some work needed to extend the authentication mechanisms Storage access: NFS / GridFTP Any kind of storage can be used as the backend... as far as it provides NFS service service GridFTP front-end to storage HSM system (NFS) FS with migration (HSM) Access Methods (SSH, HTTPs, WebDAV...) NDS Access Methods servers NDS VFS NDS logic GridFTP front-end to storage NAS appliance (NFS) GridFTP front-end to storage Any other storage(nfs) s data centres?
17 NDS features vs EUDAT (2) Persistent IDs? User always sees the same logical structure, nevertheless: The replication process: Physical location is transparent Replication process does not affect the logical namespace which Access he uses: The logical structure of VFS is the same everywhere what access method he uses: failures: The logical structure of VFS is presented similarly through different access methods As long as at least one replica is OK Pathto the file ordirectoryisconstant => Is this PID-like feature?
18 NDS features vs EUDAT (3) User-level metadata: User canassignfree-form textfilesto data objects, they can include metadata This is done through Web-GUI or procfs-like mechanism Medata search possible but not yet implemented (on the roadmap) Can above be somehow re-used in EUDAT Extendability? Functionality can be easily extended as the architecture & interfaces are open (Postgresql, NFS/GridFTP...) micro-services like approach possible: but requires effoert on the NDS consortium side For instance: some basic interfaces to meta-data can be defined (e.g. for searching data meeting some criteria) Example: We currently design and develop a mechanism for periodic data integrity checking (data scrubbing)
19 NDS2 - features Secure data storage: Data encrypted on the user side Symmetric keys to file stored in the system - protected by user s asymmetric key Integrity control on the user side MD5 s/sha1 digests stored in the system - encrypted Secure data exchange: 2-level access control: ACLs on virtual filesystem level User-side encryption and keys exchange make the sharing safe (e.g. if we don t trust provider) Secure data publication: 2 kinds of storage space: private (for internal users) and public (sanbox ed) Multiple web servers (load-balancing, HA, data synchronisation) to serve data effectively Specialised user-side tools needed: Java GUI for managing file sharing, ACLs, publication and versioning Virtual encrypted (!) filesystem for end-users: both for Linux and Windows Status: R&D project ( ); prototype expected in 2Q2013
20 Backup slides
21 NDS1: Summary Data storage & replication:: VFS level: portability and security Robust and lightweight Data replication: Automatic, transparent to users - Sync. and async modes NFS or GridFTP or GridFTP 3rd party used to access//make replicas Meta-data handling & replication: Handles file system-level medata and user-level metadata Logically centralized but DR solutions in place for quick recovery Logical filesystem structure persistency: Physical location-agnostic access Pluggable : Open interfaces, standard interfaces to external world (both user- and storage-side) We can provide custom interfaces to meta-data if needed
22 NDS architecture: (10) Meta-catalog (1) Functionality: System-level meta-data storage and handling File system structure Data replicas User-level meta-data storage Implementation: C++ library used by Data Daemon Postgres database with Slony-I replication at the backend Separation of namespaces: No sharing among users groups assumed Security by isolation Scalability multiple instances of MC for multiple users groups / insitutions Metadata DB Database
23 NDS architecture: (11) Meta-catalog (2) Meta-data redundancy: problem: reliability and performance? (1) Postgres database with Slony-I replication Each meta-catalog replicated asynchronously in master-slaves mode (Slony-I) In case of failure of master MC: slave MC is manually selected as master (DR, not full HA, human intervention needed) (2) Semi-synchronous data replication: All operations on metadata synchronously logged to distributed logs In case of failure of master MC: part of operations logged are repeated on the new master (human interv. needed) Comments: Reliability similar to synchronous DBMS replication but mechanism is lighter!!!
24 PLATON s B/A service access: sftp sftp: Well-known, secure data upload/download method WinSCP example:
25 PLATON s B/A service access: WebDAV Web Browser-based WebDAV access (read only)
26 PLATON s B/A service access: WebDAV Windows built-in WebDAV(Web Folders) client supports: mapping NDS filesystem as the Network Drive drag & drop
27 PLATON s B/A service access: NDS web application
28 PLATON s B/A service access: NDS Web application filesystem navigation
29 PLATON s B/A service access: NDS Web application meta-data view
30 PLATON s B/A service access: NDS MDFS filesystem for meta-data access
National Data Storage 2 Secure sharing, publishing and exchanging data
National Data Storage Secure sharing, publishing and exchanging data Maciej Brzeźniak, Norbert Meyer, Michał Jankowski, Gracjan Jankowski Supercomputing Department, PSNC This work is funded under National
More informationMichał Jankowski Maciej Brzeźniak PSNC
National Data Storage - architecture and mechanisms Michał Jankowski Maciej Brzeźniak PSNC Introduction Assumptions Architecture Main components Deployment Use case Agenda Data storage: The problem needs
More informationNational Data Store 2 crypto-clients - demonstration
National Data Store 2 crypto-clients - demonstration Front men : Maciej Brzeźniak, Staszek Jankowski Supercomputing Dept. of PSNC, www.psnc.pl Authors: NDS2 team at PSNC and partners full list of credits
More informationPopular backup/archival service and its application for the archival of the network traffic in the academic network PIONIER
Popular backup/archival service and its application for the archival of the network traffic in the academic network PIONIER Maciej Brzeźniak Norbert Meyer
More informationPolish National Data Storage. Norbert Meyer, Maciej Brzeźniak, Maciej Stroiński PSNC
Polish National Data Storage Norbert Meyer, Maciej Brzeźniak, Maciej Stroiński PSNC Workshop on Big Data and Open Data, Brussels. May 7-8, 2014 Data = value => needs protection! Data is value:! Expensive
More informationNDS2 Secure storage, sharing and publishing of data in the NDS
NDS2 Secure storage, sharing and publishing of data in the NDS Maciej Brzeźniak, Supercomputing Dept. of PSNC, www.psnc.pl TF-Storage meeting@dubrovnik, Sep., 26-27th 2012 Project funded by: NCBiR for
More informationNetwork Attached Storage. Jinfeng Yang Oct/19/2015
Network Attached Storage Jinfeng Yang Oct/19/2015 Outline Part A 1. What is the Network Attached Storage (NAS)? 2. What are the applications of NAS? 3. The benefits of NAS. 4. NAS s performance (Reliability
More informationFile Sharing and Network Marketing
GRAU DataSpace 2.0 THE SECURE COMMUNICATION PLATFORM FOR ENTERPRISES AND ORGANIZATIONS YOUR DATA. YOUR CONTROL Introduction Global file sharing is a mega trend Sync & Share is a new product category Consumer
More informationStorage Virtualization. Andreas Joachim Peters CERN IT-DSS
Storage Virtualization Andreas Joachim Peters CERN IT-DSS Outline What is storage virtualization? Commercial and non-commercial tools/solutions Local and global storage virtualization Scope of this presentation
More informationXtreemFS Extreme cloud file system?! Udo Seidel
XtreemFS Extreme cloud file system?! Udo Seidel Agenda Background/motivation High level overview High Availability Security Summary Distributed file systems Part of shared file systems family Around for
More informationNetwork File System (NFS) Pradipta De pradipta.de@sunykorea.ac.kr
Network File System (NFS) Pradipta De pradipta.de@sunykorea.ac.kr Today s Topic Network File System Type of Distributed file system NFS protocol NFS cache consistency issue CSE506: Ext Filesystem 2 NFS
More informationAdvancements in Storage QoS Management in National Data Storage
Advancements in Storage QoS Management in National Data Storage Darin Nikolow 1, Renata Słota 1, Stanisław Polak 1 and Jacek Kitowski 1,2 1 AGH University of Science and Technology, Faculty of Computer
More informationSmartSync NAS-to-NAS Data Replication
SmartSync NAS-to-NAS Data Replication 1. Abstract 7/23 Henry Ho Data replication has become a common feature among NAS systems. It provides a cost-effective and efficient implementation of remote data
More informationAnalisi di un servizio SRM: StoRM
27 November 2007 General Parallel File System (GPFS) The StoRM service Deployment configuration Authorization and ACLs Conclusions. Definition of terms Definition of terms 1/2 Distributed File System The
More informationCERN Cloud Storage Evaluation Geoffray Adde, Dirk Duellmann, Maitane Zotes CERN IT
SS Data & Storage CERN Cloud Storage Evaluation Geoffray Adde, Dirk Duellmann, Maitane Zotes CERN IT HEPiX Fall 2012 Workshop October 15-19, 2012 Institute of High Energy Physics, Beijing, China SS Outline
More informationETERNUS CS High End Unified Data Protection
ETERNUS CS High End Unified Data Protection Optimized Backup and Archiving with ETERNUS CS High End 0 Data Protection Issues addressed by ETERNUS CS HE 60% of data growth p.a. Rising back-up windows Too
More informationIBM TSM DISASTER RECOVERY BEST PRACTICES WITH EMC DATA DOMAIN DEDUPLICATION STORAGE
White Paper IBM TSM DISASTER RECOVERY BEST PRACTICES WITH EMC DATA DOMAIN DEDUPLICATION STORAGE Abstract This white paper focuses on recovery of an IBM Tivoli Storage Manager (TSM) server and explores
More informationSamba's Cloudy Future. Jeremy Allison Samba Team. jra@samba.org
Samba's Cloudy Future Jeremy Allison Samba Team jra@samba.org Isn't cloud storage the future? Yes, but not usable for many existing apps. Cloud Storage is a blob store Blob stores don't map very well onto
More informationArchitecture and Mode of Operation
Software- und Organisations-Service Open Source Scheduler Architecture and Mode of Operation Software- und Organisations-Service GmbH www.sos-berlin.com Scheduler worldwide Open Source Users and Commercial
More informationHadoop Distributed File System. T-111.5550 Seminar On Multimedia 2009-11-11 Eero Kurkela
Hadoop Distributed File System T-111.5550 Seminar On Multimedia 2009-11-11 Eero Kurkela Agenda Introduction Flesh and bones of HDFS Architecture Accessing data Data replication strategy Fault tolerance
More informationXtreemStore A SCALABLE STORAGE MANAGEMENT SOFTWARE WITHOUT LIMITS YOUR DATA. YOUR CONTROL
XtreemStore A SCALABLE STORAGE MANAGEMENT SOFTWARE WITHOUT LIMITS YOUR DATA. YOUR CONTROL Archive Manager - the Basis for XtreemStore DMS Email / Files ScienDfic Others PACS VIDEO PrePress CAD/CAM NFS
More information(Scale Out NAS System)
For Unlimited Capacity & Performance Clustered NAS System (Scale Out NAS System) Copyright 2010 by Netclips, Ltd. All rights reserved -0- 1 2 3 4 5 NAS Storage Trend Scale-Out NAS Solution Scaleway Advantages
More informationDistributed File Systems
Distributed File Systems Paul Krzyzanowski Rutgers University October 28, 2012 1 Introduction The classic network file systems we examined, NFS, CIFS, AFS, Coda, were designed as client-server applications.
More informationMigration and Building of Data Centers in IBM SoftLayer with the RackWare Management Module
Migration and Building of Data Centers in IBM SoftLayer with the RackWare Management Module June, 2015 WHITE PAPER Contents Advantages of IBM SoftLayer and RackWare Together... 4 Relationship between
More informationEgnyte Local Cloud Architecture. White Paper
w w w. e g n y t e. c o m Egnyte Local Cloud Architecture White Paper Revised June 21, 2012 Table of Contents Egnyte Local Cloud Introduction page 2 Scalable Solutions Personal Local Cloud page 3 Office
More informationDiagram 1: Islands of storage across a digital broadcast workflow
XOR MEDIA CLOUD AQUA Big Data and Traditional Storage The era of big data imposes new challenges on the storage technology industry. As companies accumulate massive amounts of data from video, sound, database,
More informationCollaborative SRB Data Federations
WHITE PAPER Collaborative SRB Data Federations A Unified View for Heterogeneous High-Performance Computing INTRODUCTION This paper describes Storage Resource Broker (SRB): its architecture and capabilities
More informationTop 10 Reasons why MySQL Experts Switch to SchoonerSQL - Solving the common problems users face with MySQL
SCHOONER WHITE PAPER Top 10 Reasons why MySQL Experts Switch to SchoonerSQL - Solving the common problems users face with MySQL About Schooner Information Technology Schooner Information Technology provides
More informationXtreemFS a Distributed File System for Grids and Clouds Mikael Högqvist, Björn Kolbeck Zuse Institute Berlin XtreemFS Mikael Högqvist/Björn Kolbeck 1
XtreemFS a Distributed File System for Grids and Clouds Mikael Högqvist, Björn Kolbeck Zuse Institute Berlin XtreemFS Mikael Högqvist/Björn Kolbeck 1 The XtreemOS Project Research project funded by the
More informationWe mean.network File System
We mean.network File System Introduction: Remote File-systems When networking became widely available users wanting to share files had to log in across the net to a central machine This central machine
More informationBlueArc unified network storage systems 7th TF-Storage Meeting. Scale Bigger, Store Smarter, Accelerate Everything
BlueArc unified network storage systems 7th TF-Storage Meeting Scale Bigger, Store Smarter, Accelerate Everything BlueArc s Heritage Private Company, founded in 1998 Headquarters in San Jose, CA Highest
More informationOpen Source, Scale-out clustered NAS using nfs-ganesha and GlusterFS
Open Source, Scale-out clustered NAS using nfs-ganesha and GlusterFS Anand Subramanian Senior Principal Engineer, Red Hat anands@redhat.com Agenda Introduction GlusterFS NFSv4 nfs-ganesha Nfs-ganesha Architecture
More informationDirect NFS - Design considerations for next-gen NAS appliances optimized for database workloads Akshay Shah Gurmeet Goindi Oracle
Direct NFS - Design considerations for next-gen NAS appliances optimized for database workloads Akshay Shah Gurmeet Goindi Oracle Agenda Introduction Database Architecture Direct NFS Client NFS Server
More informationHigh Availability Solutions for the MariaDB and MySQL Database
High Availability Solutions for the MariaDB and MySQL Database 1 Introduction This paper introduces recommendations and some of the solutions used to create an availability or high availability environment
More informationMANAGEMENT METHODS IN SLA-AWARE DISTRIBUTED STORAGE SYSTEMS
Computer Science 13 (3) 2012 http://dx.doi.org/10.7494/csci.2012.13.3.35 Darin Nikolow Renata S lota Danilo Lakovic Pawe l Winiarczyk Marek Pogoda Jacek Kitowski MANAGEMENT METHODS IN SLA-AWARE DISTRIBUTED
More informationMigration and Building of Data Centers in IBM SoftLayer with the RackWare Management Module
Migration and Building of Data Centers in IBM SoftLayer with the RackWare Management Module June, 2015 WHITE PAPER Contents Advantages of IBM SoftLayer and RackWare Together... 4 Relationship between
More informationLong term retention and archiving the challenges and the solution
Long term retention and archiving the challenges and the solution NAME: Yoel Ben-Ari TITLE: VP Business Development, GH Israel 1 Archive Before Backup EMC recommended practice 2 1 Backup/recovery process
More informationHigh-Availability Using Open Source Software
High-Availability Using Open Source Software Luka Perkov Iskon Internet, Zagreb, Croatia Nikola Pavković Ruđer Bošković Institute Bijenička cesta Zagreb, Croatia Juraj Petrović Faculty of Electrical Engineering
More informationCERNBox + EOS: Cloud Storage for Science
Data & Storage Services CERNBox + EOS: Cloud Storage for Science CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/it Presenter: Luca Masce. Thanks to: Jakub T. Mościcki, Andreas J. Peters,
More informationTHE HADOOP DISTRIBUTED FILE SYSTEM
THE HADOOP DISTRIBUTED FILE SYSTEM Konstantin Shvachko, Hairong Kuang, Sanjay Radia, Robert Chansler Presented by Alexander Pokluda October 7, 2013 Outline Motivation and Overview of Hadoop Architecture,
More informationSURFsara Data Services
SURFsara Data Services SUPPORTING DATA-INTENSIVE SCIENCES Mark van de Sanden The world of the many Many different users (well organised (international) user communities, research groups, universities,
More informationData Management in an International Data Grid Project. Timur Chabuk 04/09/2007
Data Management in an International Data Grid Project Timur Chabuk 04/09/2007 Intro LHC opened in 2005 several Petabytes of data per year data created at CERN distributed to Regional Centers all over the
More informationCloud Based Application Architectures using Smart Computing
Cloud Based Application Architectures using Smart Computing How to Use this Guide Joyent Smart Technology represents a sophisticated evolution in cloud computing infrastructure. Most cloud computing products
More informationThe Future of PostgreSQL High Availability Robert Hodges - Continuent, Inc. Simon Riggs - 2ndQuadrant
The Future of PostgreSQL High Availability Robert Hodges - Continuent, Inc. Simon Riggs - 2ndQuadrant Agenda / Introductions / Framing the High Availability (HA) Problem / Hot Standby + Log Streaming /
More informationNAS 259 Protecting Your Data with Remote Sync (Rsync)
NAS 259 Protecting Your Data with Remote Sync (Rsync) Create and execute an Rsync backup job A S U S T O R C O L L E G E COURSE OBJECTIVES Upon completion of this course you should be able to: 1. Having
More informationDESYcloud: an owncloud & dcache update
: an owncloud & dcache update Paul Millar (on behalf of team) : an owncloud & dcache update Cloud Services for Synchronisation and Sharing Zürich, Switzerland. 2016-01-18 2016-01-19 http://cs3.ethz.ch/
More informationTake An Internal Look at Hadoop. Hairong Kuang Grid Team, Yahoo! Inc hairong@yahoo-inc.com
Take An Internal Look at Hadoop Hairong Kuang Grid Team, Yahoo! Inc hairong@yahoo-inc.com What s Hadoop Framework for running applications on large clusters of commodity hardware Scale: petabytes of data
More informationSoftware to Simplify and Share SAN Storage Sanbolic s SAN Storage Enhancing Software Portfolio
Software to Simplify and Share SAN Storage Sanbolic s SAN Storage Enhancing Software Portfolio www.sanbolic.com Table of Contents About Sanbolic... 3 Melio File System... 3 LaScala Volume Manager... 3
More informationDistributed File System Choices: Red Hat Storage, GFS2 & pnfs
Distributed File System Choices: Red Hat Storage, GFS2 & pnfs Ric Wheeler Architect & Senior Manager, Red Hat June 27, 2012 Overview Distributed file system basics Red Hat distributed file systems Performance
More informationIntro to AWS: Storage Services
Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect 2015, Amazon Web Services, Inc. or its affiliates. All rights reserved AWS storage options Scalable object storage Inexpensive archive
More informationVMware vsphere Data Protection
VMware vsphere Data Protection Replication Target TECHNICAL WHITEPAPER 1 Table of Contents Executive Summary... 3 VDP Identities... 3 vsphere Data Protection Replication Target Identity (VDP-RT)... 3 Replication
More informationIBM Tivoli Storage Manager Version 7.1.4. Introduction to Data Protection Solutions IBM
IBM Tivoli Storage Manager Version 7.1.4 Introduction to Data Protection Solutions IBM IBM Tivoli Storage Manager Version 7.1.4 Introduction to Data Protection Solutions IBM Note: Before you use this
More informationDSS. High performance storage pools for LHC. Data & Storage Services. Łukasz Janyst. on behalf of the CERN IT-DSS group
DSS High performance storage pools for LHC Łukasz Janyst on behalf of the CERN IT-DSS group CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it Introduction The goal of EOS is to provide a
More informationIntroduction to Gluster. Versions 3.0.x
Introduction to Gluster Versions 3.0.x Table of Contents Table of Contents... 2 Overview... 3 Gluster File System... 3 Gluster Storage Platform... 3 No metadata with the Elastic Hash Algorithm... 4 A Gluster
More informationConstant Replicator: An Introduction
Data Availability Storage Software Constant Replicator: An Introduction Configurations, Applications and Solutions of Constant Replicator A White Paper A Constant Data Technology Document August 2004 Copyright
More informationowncloud Architecture Overview
owncloud Architecture Overview owncloud, Inc. 57 Bedford Street, Suite 102 Lexington, MA 02420 United States phone: +1 (877) 394-2030 www.owncloud.com/contact owncloud GmbH Schloßäckerstraße 26a 90443
More informationService Overview CloudCare Online Backup
Service Overview CloudCare Online Backup CloudCare s Online Backup service is a secure, fully automated set and forget solution, powered by Attix5, and is ideal for organisations with limited in-house
More information<Insert Picture Here> Oracle Cloud Storage. Morana Kobal Butković Principal Sales Consultant Oracle Hrvatska
Oracle Cloud Storage Morana Kobal Butković Principal Sales Consultant Oracle Hrvatska Oracle Cloud Storage Automatic Storage Management (ASM) Oracle Cloud File System ASM Dynamic
More informationTushar Joshi Turtle Networks Ltd
MySQL Database for High Availability Web Applications Tushar Joshi Turtle Networks Ltd www.turtle.net Overview What is High Availability? Web/Network Architecture Applications MySQL Replication MySQL Clustering
More informationImplementing the Hadoop Distributed File System Protocol on OneFS Jeff Hughes EMC Isilon
Implementing the Hadoop Distributed File System Protocol on OneFS Jeff Hughes EMC Isilon Outline Hadoop Overview OneFS Overview MapReduce + OneFS Details of isi_hdfs_d Wrap up & Questions 2 Hadoop Overview
More informationPerformance, Reliability, and Operational Issues for High Performance NAS Storage on Cray Platforms. Cray User Group Meeting June 2007
Performance, Reliability, and Operational Issues for High Performance NAS Storage on Cray Platforms Cray User Group Meeting June 2007 Cray s Storage Strategy Background Broad range of HPC requirements
More informationLevel 1: Asigra Cloud Backup Foundation Training
Level 1: Asigra Cloud Backup Foundation Training Course Description Delivery Method: Virtual Instructor Led Training Duration: 3 Days Course Objectives: This course teaches the fundamental operations of
More informationdcache, Software for Big Data
dcache, Software for Big Data Innovation Day 2013, Berlin Patrick Fuhrmann dcache Innovation Day Berlin Patrick Fuhrmann 10 December 2013 1 About Technology and further roadmap Collaboration and partners
More informationRemote File System Suite
Remote File System Suite Softwarepraktikum für Fortgeschrittene Michael Kuhn Parallele und Verteilte Systeme Institut für Informatik Ruprecht-Karls-Universität Heidelberg 2009-07-07 1 / 22 1 Introduction
More informationData Replication INSTALATION GUIDE. Open-E Data Storage Server (DSS ) Integrated Data Replication reduces business downtime.
Open-E Data Storage Server (DSS ) Data Replication INSTALATION GUIDE Integrated Data Replication reduces business downtime. Open-E DSS Data Replication Open-E Data Server Storage (DSS) offers enterprise-class
More informationHDFS Under the Hood. Sanjay Radia. Sradia@yahoo-inc.com Grid Computing, Hadoop Yahoo Inc.
HDFS Under the Hood Sanjay Radia Sradia@yahoo-inc.com Grid Computing, Hadoop Yahoo Inc. 1 Outline Overview of Hadoop, an open source project Design of HDFS On going work 2 Hadoop Hadoop provides a framework
More informationDeployment Topologies
, page 1 Multinode Cluster with Unified Nodes, page 2 Clustering Considerations, page 3 Cisco Unified Communications Domain Manager 10.6(x) Redundancy and Disaster Recovery, page 4 Capacity Considerations,
More informationProtect Microsoft Exchange databases, achieve long-term data retention
Technical white paper Protect Microsoft Exchange databases, achieve long-term data retention HP StoreOnce Backup systems, HP StoreOnce Catalyst, and Symantec NetBackup OpenStorage Table of contents Introduction...
More informationDigital Library for Multimedia Content Management
Digital Library for Multimedia Content Management Cezary Mazurek, Maciej Stroinski, Sebastian Szuber Pozna_ Supercomputing and Networking Centre, ul. Noskowskiego 10, 61-704 Pozna_, POLAND tel. +48 61
More informationAvailability Digest. www.availabilitydigest.com. Redundant Load Balancing for High Availability July 2013
the Availability Digest Redundant Load Balancing for High Availability July 2013 A large data center can comprise hundreds or thousands of servers. These servers must not only be interconnected, but they
More informationIntegrating Content Management Within Enterprise Applications: The Open Standards Option. Copyright Xythos Software, Inc. 2005 All Rights Reserved
Integrating Content Management Within Enterprise Applications: The Open Standards Option Copyright Xythos Software, Inc. 2005 All Rights Reserved Table of Contents Introduction...3 Why Developers Are Choosing
More informationDeploying Silver Peak VXOA with EMC Isilon SyncIQ. February 2012. www.silver-peak.com
Deploying Silver Peak VXOA with EMC Isilon SyncIQ February 2012 www.silver-peak.com Table of Contents Table of Contents Overview... 3 Solution Components... 3 EMC Isilon...3 Isilon SyncIQ... 3 Silver Peak
More informationBookKeeper. Flavio Junqueira Yahoo! Research, Barcelona. Hadoop in China 2011
BookKeeper Flavio Junqueira Yahoo! Research, Barcelona Hadoop in China 2011 What s BookKeeper? Shared storage for writing fast sequences of byte arrays Data is replicated Writes are striped Many processes
More informationHow To Improve Afs.Org For Free On A Pc Or Mac Or Ipad (For Free) For A Long Time (For A Long Term Time) For Free (For Cheap) For Your Computer Or Your Hard Drive) For The Long
Next Generation Storage Clouds Jeffrey Altman, President Your File System Inc. 29 September 2009 OpenAFS Roadmap? Or Wish List? At every Workshop and Conference a roadmap is presented but its not a roadmap
More informationTHE EXPAND PARALLEL FILE SYSTEM A FILE SYSTEM FOR CLUSTER AND GRID COMPUTING. José Daniel García Sánchez ARCOS Group University Carlos III of Madrid
THE EXPAND PARALLEL FILE SYSTEM A FILE SYSTEM FOR CLUSTER AND GRID COMPUTING José Daniel García Sánchez ARCOS Group University Carlos III of Madrid Contents 2 The ARCOS Group. Expand motivation. Expand
More informationBig data management with IBM General Parallel File System
Big data management with IBM General Parallel File System Optimize storage management and boost your return on investment Highlights Handles the explosive growth of structured and unstructured data Offers
More informationPrivate Cloud Storage for Media Applications. Bang Chang Vice President, Broadcast Servers and Storage bang.chang@xor-media.com
Private Cloud Storage for Media Bang Chang Vice President, Broadcast Servers and Storage bang.chang@xor-media.com Table of Contents Introduction Cloud Storage Requirements Application transparency Universal
More informationEMC IRODS RESOURCE DRIVERS
EMC IRODS RESOURCE DRIVERS PATRICK COMBES: PRINCIPAL SOLUTION ARCHITECT, LIFE SCIENCES 1 QUICK AGENDA Intro to Isilon (~2 hours) Isilon resource driver Intro to ECS (~1.5 hours) ECS Resource driver Possibilities
More informationTechnical. Overview. ~ a ~ irods version 4.x
Technical Overview ~ a ~ irods version 4.x The integrated Ru e-oriented DATA System irods is open-source, data management software that lets users: access, manage, and share data across any type or number
More informationWOS Cloud. ddn.com. Personal Storage for the Enterprise. DDN Solution Brief
DDN Solution Brief Personal Storage for the Enterprise WOS Cloud Secure, Shared Drop-in File Access for Enterprise Users, Anytime and Anywhere 2011 DataDirect Networks. All Rights Reserved DDN WOS Cloud
More informationThe dcache Storage Element
16. Juni 2008 Hamburg The dcache Storage Element and it's role in the LHC era for the dcache team Topics for today Storage elements (SEs) in the grid Introduction to the dcache SE Usage of dcache in LCG
More informationFAN An Architecture for Scalable, Service-Oriented Data Management
FAN An Architecture for Scalable, Service-Oriented Data Management Richard Gillett Acopia Networks SNIA Legal Notice The material contained in this tutorial is copyrighted by the SNIA. Member companies
More informationReplication Security
Replication Security PGConf NYC, 2014 New York City, NY Magnus Hagander magnus@hagander.net PRODUCTS CONSULTING APPLICATION MANAGEMENT IT OPERATIONS SUPPORT TRAINING Magnus Hagander PostgreSQL Core Team
More informationIntroduction to Highly Available NFS Server on scale out storage systems based on GlusterFS
Introduction to Highly Available NFS Server on scale out storage systems based on GlusterFS Soumya Koduri Red Hat Meghana Madhusudhan Red Hat AGENDA What is GlusterFS? Integration with NFS Ganesha Clustered
More informationOpen Source Cloud Computing Management with OpenNebula
CloudCamp Campus Party July 2011, Valencia Open Source Cloud Computing Management with OpenNebula Javier Fontán Muiños dsa-research.org Distributed Systems Architecture Research Group Universidad Complutense
More informationList of Figures and Tables
List of Figures and Tables FIGURES 1.1 Server-Centric IT architecture 2 1.2 Inflexible allocation of free storage capacity 3 1.3 Storage-Centric IT architecture 4 1.4 Server upgrade: preparation of a new
More informationXtreemFS - a distributed and replicated cloud file system
XtreemFS - a distributed and replicated cloud file system Michael Berlin Zuse Institute Berlin DESY Computing Seminar, 16.05.2011 Who we are Zuse Institute Berlin operates the HLRN supercomputer (#63+64)
More informationEMC DATA DOMAIN OVERVIEW. Copyright 2011 EMC Corporation. All rights reserved.
EMC DATA DOMAIN OVERVIEW 1 2 With Data Domain Deduplication Storage Systems, You Can WAN Retain longer Keep backups onsite longer with less disk for fast, reliable restores, and eliminate the use of tape
More informationCaching SMB Data for Offline Access and an Improved Online Experience
Caching SMB Data for Offline Access and an Improved Online Experience Agenda What is Offline Files How does Offline Files interact with SMB Offline Files enhancements for Windows 7 Questions 2 What is
More informationMIGRATING DESKTOP AND ROAMING ACCESS. Migrating Desktop and Roaming Access Whitepaper
Migrating Desktop and Roaming Access Whitepaper Poznan Supercomputing and Networking Center Noskowskiego 12/14 61-704 Poznan, POLAND 2004, April white-paper-md-ras.doc 1/11 1 Product overview In this whitepaper
More informationWOS OBJECT STORAGE PRODUCT BROCHURE DDN.COM 1.800.837.2298. 360 Full Spectrum Object Storage
PRODUCT BROCHURE WOS OBJECT STORAGE 360 Full Spectrum Object Storage The promise of object storage is simple: to enable organizations to build highly Performance Scalability Reliability Efficiency Security
More informationCompetitive Analysis Retrospect And Our Competition
Competitive Analysis And Our Competition September 2013 Competitive Analysis: for Windows and our top competitors We ve taken a hard look at our competition and how we stack up. Here is a comparison of
More informationDisaster Recovery for Oracle Database
Disaster Recovery for Oracle Database Zero Data Loss Recovery Appliance, Active Data Guard and Oracle GoldenGate ORACLE WHITE PAPER APRIL 2015 Overview Oracle Database provides three different approaches
More informationMichael Thomas, Dorian Kcira California Institute of Technology. CMS Offline & Computing Week
Michael Thomas, Dorian Kcira California Institute of Technology CMS Offline & Computing Week San Diego, April 20-24 th 2009 Map-Reduce plus the HDFS filesystem implemented in java Map-Reduce is a highly
More informationScientific Storage at FNAL. Gerard Bernabeu Altayo Dmitry Litvintsev Gene Oleynik 14/10/2015
Scientific Storage at FNAL Gerard Bernabeu Altayo Dmitry Litvintsev Gene Oleynik 14/10/2015 Index - Storage use cases - Bluearc - Lustre - EOS - dcache disk only - dcache+enstore Data distribution by solution
More informationEMC DATA PROTECTION. Backup ed Archivio su cui fare affidamento
EMC DATA PROTECTION Backup ed Archivio su cui fare affidamento 1 Challenges with Traditional Tape Tightening backup windows Lengthy restores Reliability, security and management issues Inability to meet
More informationDesigning a Cloud Storage System
Designing a Cloud Storage System End to End Cloud Storage When designing a cloud storage system, there is value in decoupling the system s archival capacity (its ability to persistently store large volumes
More informationCurrent Status of FEFS for the K computer
Current Status of FEFS for the K computer Shinji Sumimoto Fujitsu Limited Apr.24 2012 LUG2012@Austin Outline RIKEN and Fujitsu are jointly developing the K computer * Development continues with system
More informationovirt and Gluster Hyperconvergence
ovirt and Gluster Hyperconvergence January 2015 Federico Simoncelli Principal Software Engineer Red Hat ovirt and GlusterFS Hyperconvergence, Jan 2015 1 Agenda ovirt Architecture and Software-defined Data
More informationOverview. Big Data in Apache Hadoop. - HDFS - MapReduce in Hadoop - YARN. https://hadoop.apache.org. Big Data Management and Analytics
Overview Big Data in Apache Hadoop - HDFS - MapReduce in Hadoop - YARN https://hadoop.apache.org 138 Apache Hadoop - Historical Background - 2003: Google publishes its cluster architecture & DFS (GFS)
More information