High Performance Computing Specialists. ZFS Storage as a Solution for Big Data and Flexibility

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "High Performance Computing Specialists. ZFS Storage as a Solution for Big Data and Flexibility"

Transcription

1 High Performance Computing Specialists ZFS Storage as a Solution for Big Data and Flexibility

2 Introducing VA Technologies UK Based System Integrator Specialising in High Performance ZFS Storage Partner of E4 Computing Engineering Delivering ZFS Storage and HPC Solutions

3 VA HPC Powered by E4 New HPC Solutions ARKA ARM and Quadro + Tegra 3 Blades Joint New Solutions Lustre on ZFS & Hadoop on ZFS

4 But First. ZFS 6 Great Reasons to Love ZFS Architecture Data Integrity Redundancy Transactional Copy on Write (COW) Snapshots Hybrid Storage Mixing SSD with HDD

5 Architecture 6 Great Reasons to Love ZFS Architecture Data Integrity Redundancy Transactional Copy on Write (COW) Snapshots Hybrid Storage Mixing SSD with HDD

6 Architecture ZFS Pool Layout Clones Pool Configuration zvol Information zvol File System zvol File System Snapshots

7 Architecture ZFS Layer View raw swapdumpiscsi?? ZFS NFS CIFS?? ZFS Volume Emulator (Zvol) ZFS POSIX Layer (ZPL) e.g. pnfs?? Transactional Object Layer Pooled Storage Layer Block Device Driver HDD SSD iscsi FC??

8 Data Integrity 5 Great Reasons to Love ZFS Architecture Data Integrity Redundancy Transactional Copy on Write (COW) Snapshots Hybrid Storage Mixing SSD with HDD

9 Data Integrity Designers Quote The job of any file system boils down to this: when asked to read a block, it should return the same data that was previously written to that block. If it can't do that -- because the disk is offline or the data has been damaged or tampered with -- it should detect this and return an error. Jeff Bonwick Father of ZFS Dec 2008

10 Data Integrity Merkle Trees & Checksums

11 Data Integrity Validating Checksum

12 Data Integrity Do you have a Write Hole?

13 Redundancy 6 Great Reasons to Love ZFS Architecture Data Integrity Redundancy Transactional Copy on Write (COW) Snapshots Hybrid Storage Mixing SSD with HDD

14 Redundancy Mirrored Disks in 2 vdevs root vdev Logical vdevs top-level vdev children[0] mirror top-level vdev children[1] mirror vdev type = disk children[0] Vdev vdev type = disk type = disk children[0] children[0] Physical or leaf vdevs vdev type = disk children[0]

15 Redundancy RAIDz2 in 3vdevs vdev-0 RAID-Z2 HDD HDD HDD HDD HDD HDD zpool vdev-1 RAID-Z2 HDD HDD HDD HDD HDD HDD vdev-x RAID-Z2 HDD HDD HDD HDD HDD HDD Physical or leaf vdevs

16 Redundancy Dynamic Striping RAID-0 Column size = 128 kbytes, stripe width = 384 kbytes 384 kbytes ZFS Dynamic Stripe recordsize = 128 kbytes vdevs Total write size = 2816 kbytes

17 Transactional Copy on Write (COW) 6 Great Reasons to Love ZFS Architecture Data Integrity Redundancy Transactional Copy on Write (COW) Snapshots Hybrid Storage Mixing SSD with HDD

18 Transactional Copy on Write (COW) Journaling FSCK

19 1. Initial block tree 2. COW some data 3. COW metadata 4. Update Uberblocks & free

20 Snapshots 6 Great Reasons to Love ZFS Architecture Data Integrity Redundancy Transactional Copy on Write (COW) Snapshots Hybrid Storage Mixing SSD with HDD

21 Snapshots

22 Hybrid Storage Pools Mixing SSD with HDD 6 Great Reasons to Love ZFS Architecture Data Integrity Redundancy Transactional Copy on Write (COW) Snapshots Hybrid Storage Mixing SSD with HDD

23 Hybrid Storage Pools Mixing SSD with HDD Old Old New ZFS

24 Hybrid Storage Pools Mixing SSD with HDD Adaptive Replacement Cache (ARC) Separate ZFS Intent Log (ZIL) Main Pool Level 2 ARC (L2ARC) Write optimized device (SSD) HDD HDD HDD Read optimized device (SSD)

25 Hybrid Storage Pools ARC Evict the oldest single-use entry Miss Hit LRU Recent Cache MRU MFU Frequent Cache LFU RAM Evict the oldest multiple accessed entry

26 Hybrid Storage Pools L2ARC Data soon to be evicted from the ARC sent to the Layer 2 ARC (L2ARC) which is usually a SSD vdev Works well when cache vdev is optimized for fast reads lower latency than pool disks inexpensive way to read performance SSD vdev can be striped for better performance Non Persistent Requires a rebuild after power off (soon to change!) ARC data soon to be evicted cache

27 Hybrid Storage Pools ZIL/SLOG Known as the Write Cache Non Volatile Stores small (<32k Configurable ) sync writes in high speed persistent storage or a separate log device (SLOG) Flushes to disk backend periodically sequential write stream as part of the TXG group Assigned on a per pool basis Perfect use case for RAM Based SAS Devices

28 ZFS Limitations Parallel Access? Distributed File System? pnfs

29 Relevancy to the HPC Community

30 Hadoop on ZFS Rebuild Times Utilize SSD s with HDD Better Administration Single Drive Replacement Integrated Management Warning and Reporting

31 Linux and Lustre on ZFS Native Kernel Module now available.

32 Linux and Lustre on ZFS 55PB Now Active to Sequoia Users Lustre + ZFS Fully Configured 768 OSS & OST s

33

34 Lustre on ZFS Write Performance Single shared file IOR (10G block, 1M transfers) Sequoia Workload 768 OSS Nodes 2048 Tasks per OSS 1,572,864 Compute Cores MB/s LDISKFS Increased tasks per OSS degrades performance ZFS - Constant performance Increase I/O size for RAIDZ2 LDISKFS+RAI D6 ZFS+RAID6 ZFS+RAIDZ2

35 Lustre on ZFS Read Performance LDISKFS mballocallowslargeri/o ZFS 128Kmaximumblocksize IOPs limited for ZFS+RAID6 Perfect Opportunity for Read Caching LDISKFS+RAI D6 ZFS+RAID6 ZFS+RAIDZ2

36 Lustre on ZFS Coming Soon New Hardware Optimised for Lustre on ZFS Low Power Consumption OSS & OST Customisable for your Lustre deployment Full Lustre and HW Support

37 Thanks Very Much! Ryan Tyler VA Technologies

ZFS In Business. Roch Bourbonnais Sun Microsystems Roch.Bourbonnais@sun.com

ZFS In Business. Roch Bourbonnais Sun Microsystems Roch.Bourbonnais@sun.com ZFS In Business Roch Bourbonnais Sun Microsystems Roch.Bourbonnais@sun.com 1 What is ZFS Integrated Volume and Filesystem w no predefined limits Volume Management > pooling of disks, luns... in raid-z

More information

White Paper. Educational. Measuring Storage Performance

White Paper. Educational. Measuring Storage Performance TABLE OF CONTENTS Introduction....... Storage Performance Metrics.... Factors Affecting Storage Performance....... Provisioning IOPS in Hardware-Defined Solutions....... Provisioning IOPS in Software-Defined

More information

Practical Applications of Lustre/ZFS Hybrid Systems LUG 2014 Miami FL

Practical Applications of Lustre/ZFS Hybrid Systems LUG 2014 Miami FL Practical Applications of Lustre/ZFS Hybrid Systems LUG 2014 Miami FL Q2-2014 Josh Judd, CTO Agenda Brief Review: Luster over ZFS Brief Overview: platforms used in example solutions Discuss three cases

More information

NexentaStor Enterprise Backend for CLOUD. Marek Lubinski Marek Lubinski Sr VMware/Storage Engineer, LeaseWeb B.V.

NexentaStor Enterprise Backend for CLOUD. Marek Lubinski Marek Lubinski Sr VMware/Storage Engineer, LeaseWeb B.V. NexentaStor Enterprise Backend for CLOUD Marek Lubinski Marek Lubinski Sr VMware/Storage Engineer, LeaseWeb B.V. AGENDA LeaseWeb overview Express Cloud platform Initial storage build Why NexentaStor NexentaStor

More information

ZFS Administration 1

ZFS Administration 1 ZFS Administration 1 With a rapid paradigm-shift towards digital content and large datasets, managing large amounts of data can be a challenging task. Before implementing a storage solution, there are

More information

NEXENTA S VDI SOLUTIONS BRAD STONE GENERAL MANAGER NEXENTA GREATERCHINA

NEXENTA S VDI SOLUTIONS BRAD STONE GENERAL MANAGER NEXENTA GREATERCHINA NEXENTA S VDI SOLUTIONS BRAD STONE GENERAL MANAGER NEXENTA GREATERCHINA VDI Storage Challenge 95% of I/O is small, random writes Very challenging for a storage system End users demand low latency NexentaStor

More information

SOLUTION BRIEF. Resolving the VDI Storage Challenge

SOLUTION BRIEF. Resolving the VDI Storage Challenge CLOUDBYTE ELASTISTOR QOS GUARANTEE MEETS USER REQUIREMENTS WHILE REDUCING TCO The use of VDI (Virtual Desktop Infrastructure) enables enterprises to become more agile and flexible, in tune with the needs

More information

New Storage System Solutions

New Storage System Solutions New Storage System Solutions Craig Prescott Research Computing May 2, 2013 Outline } Existing storage systems } Requirements and Solutions } Lustre } /scratch/lfs } Questions? Existing Storage Systems

More information

Performance, Reliability, and Operational Issues for High Performance NAS Storage on Cray Platforms. Cray User Group Meeting June 2007

Performance, Reliability, and Operational Issues for High Performance NAS Storage on Cray Platforms. Cray User Group Meeting June 2007 Performance, Reliability, and Operational Issues for High Performance NAS Storage on Cray Platforms Cray User Group Meeting June 2007 Cray s Storage Strategy Background Broad range of HPC requirements

More information

HOW TRUENAS LEVERAGES OPENZFS. Storage and Servers Driven by Open Source. marketing@ixsystems.com

HOW TRUENAS LEVERAGES OPENZFS. Storage and Servers Driven by Open Source. marketing@ixsystems.com HOW TRUENAS LEVERAGES OPENZFS Storage and Servers Driven by Open Source marketing@ixsystems.com CONTENTS 1 Executive Summary 2 History of ixsystems 3 Overview of TrueNAS 4 OpenZFS 4.1 History 4.2 Technical

More information

Workload Dependent Performance Evaluation of the Btrfs and ZFS Filesystems

Workload Dependent Performance Evaluation of the Btrfs and ZFS Filesystems Workload Dependent Performance Evaluation of the Btrfs and ZFS Dominique A. Heger DHTechnologies, Austin, Texas dheger@dhtusa.com. Introduction The UNIX and Linux operating systems alike already provide

More information

High Performance Computing OpenStack Options. September 22, 2015

High Performance Computing OpenStack Options. September 22, 2015 High Performance Computing OpenStack PRESENTATION TITLE GOES HERE Options September 22, 2015 Today s Presenters Glyn Bowden, SNIA Cloud Storage Initiative Board HP Helion Professional Services Alex McDonald,

More information

Flash Performance in Storage Systems. Bill Moore Chief Engineer, Storage Systems Sun Microsystems

Flash Performance in Storage Systems. Bill Moore Chief Engineer, Storage Systems Sun Microsystems Flash Performance in Storage Systems Bill Moore Chief Engineer, Storage Systems Sun Microsystems 1 Disk to CPU Discontinuity Moore s Law is out-stripping disk drive performance (rotational speed) As a

More information

Sun Storage Perspective & Lustre Architecture. Dr. Peter Braam VP Sun Microsystems

Sun Storage Perspective & Lustre Architecture. Dr. Peter Braam VP Sun Microsystems Sun Storage Perspective & Lustre Architecture Dr. Peter Braam VP Sun Microsystems Agenda Future of Storage Sun s vision Lustre - vendor neutral architecture roadmap Sun s view on storage introduction The

More information

DEPLOYING HYBRID STORAGE POOLS With Sun Flash Technology and the Solaris ZFS File System. Roger Bitar, Sun Microsystems. Sun BluePrints Online

DEPLOYING HYBRID STORAGE POOLS With Sun Flash Technology and the Solaris ZFS File System. Roger Bitar, Sun Microsystems. Sun BluePrints Online DEPLOYING HYBRID STORAGE POOLS With Sun Flash Technology and the Solaris ZFS File System Roger Bitar, Sun Microsystems Sun BluePrints Online Part No 820-5881-10 Revision 1.0, 10/31/08 Sun Microsystems,

More information

The Use of Flash in Large-Scale Storage Systems. Nathan.Rutman@Seagate.com

The Use of Flash in Large-Scale Storage Systems. Nathan.Rutman@Seagate.com The Use of Flash in Large-Scale Storage Systems Nathan.Rutman@Seagate.com 1 Seagate s Flash! Seagate acquired LSI s Flash Components division May 2014 Selling multiple formats / capacities today Nytro

More information

WARP Mechanics PetaPod and TeraPod Series Analytics Engines

WARP Mechanics PetaPod and TeraPod Series Analytics Engines WARP Mechanics PetaPod and TeraPod Series Analytics Engines Features & Benefits Line of turnkey Hadoop analytics appliances Ease of deployment and management at all layers Super computer (HPC) class storage

More information

Filesystems Performance in GNU/Linux Multi-Disk Data Storage

Filesystems Performance in GNU/Linux Multi-Disk Data Storage JOURNAL OF APPLIED COMPUTER SCIENCE Vol. 22 No. 2 (2014), pp. 65-80 Filesystems Performance in GNU/Linux Multi-Disk Data Storage Mateusz Smoliński 1 1 Lodz University of Technology Faculty of Technical

More information

Cloud Storage. Parallels. Performance Benchmark Results. White Paper. www.parallels.com

Cloud Storage. Parallels. Performance Benchmark Results. White Paper. www.parallels.com Parallels Cloud Storage White Paper Performance Benchmark Results www.parallels.com Table of Contents Executive Summary... 3 Architecture Overview... 3 Key Features... 4 No Special Hardware Requirements...

More information

Analyzing Big Data with Splunk A Cost Effective Storage Architecture and Solution

Analyzing Big Data with Splunk A Cost Effective Storage Architecture and Solution Analyzing Big Data with Splunk A Cost Effective Storage Architecture and Solution Jonathan Halstuch, COO, RackTop Systems JHalstuch@racktopsystems.com Big Data Invasion We hear so much on Big Data and

More information

VNX HYBRID FLASH BEST PRACTICES FOR PERFORMANCE

VNX HYBRID FLASH BEST PRACTICES FOR PERFORMANCE 1 VNX HYBRID FLASH BEST PRACTICES FOR PERFORMANCE JEFF MAYNARD, CORPORATE SYSTEMS ENGINEER 2 ROADMAP INFORMATION DISCLAIMER EMC makes no representation and undertakes no obligations with regard to product

More information

Jumpstart VDI Deployments with NexentaVSA for View

Jumpstart VDI Deployments with NexentaVSA for View Jumpstart VDI Deployments with NexentaVSA for View White Paper March 2012 Jumpstart VDI Deployments with NexentaVSA for View Table of Contents Challenges in Deploying VDI... 1 Leveraging NexentaStor...

More information

Best Practices for Deploying Citrix XenDesktop on NexentaStor Open Storage

Best Practices for Deploying Citrix XenDesktop on NexentaStor Open Storage Best Practices for Deploying Citrix XenDesktop on NexentaStor Open Storage White Paper July, 2011 Deploying Citrix XenDesktop on NexentaStor Open Storage Table of Contents The Challenges of VDI Storage

More information

Getting performance & scalability on standard platforms, the Object vs Block storage debate. Copyright 2013 MPSTOR LTD. All rights reserved.

Getting performance & scalability on standard platforms, the Object vs Block storage debate. Copyright 2013 MPSTOR LTD. All rights reserved. Getting performance & scalability on standard platforms, the Object vs Block storage debate 1 December Webinar Session Getting performance & scalability on standard platforms, the Object vs Block storage

More information

StorPool Distributed Storage Software Technical Overview

StorPool Distributed Storage Software Technical Overview StorPool Distributed Storage Software Technical Overview StorPool 2015 Page 1 of 8 StorPool Overview StorPool is distributed storage software. It pools the attached storage (hard disks or SSDs) of standard

More information

Delivering SDS simplicity and extreme performance

Delivering SDS simplicity and extreme performance Delivering SDS simplicity and extreme performance Real-World SDS implementation of getting most out of limited hardware Murat Karslioglu Director Storage Systems Nexenta Systems October 2013 1 Agenda Key

More information

Netapp HPC Solution for Lustre. Rich Fenton (fenton@netapp.com) UK Solutions Architect

Netapp HPC Solution for Lustre. Rich Fenton (fenton@netapp.com) UK Solutions Architect Netapp HPC Solution for Lustre Rich Fenton (fenton@netapp.com) UK Solutions Architect Agenda NetApp Introduction Introducing the E-Series Platform Why E-Series for Lustre? Modular Scale-out Capacity Density

More information

PARALLELS CLOUD STORAGE

PARALLELS CLOUD STORAGE PARALLELS CLOUD STORAGE Performance Benchmark Results 1 Table of Contents Executive Summary... Error! Bookmark not defined. Architecture Overview... 3 Key Features... 5 No Special Hardware Requirements...

More information

Current Status of FEFS for the K computer

Current Status of FEFS for the K computer Current Status of FEFS for the K computer Shinji Sumimoto Fujitsu Limited Apr.24 2012 LUG2012@Austin Outline RIKEN and Fujitsu are jointly developing the K computer * Development continues with system

More information

White Paper. NexentaConnect TM Technology Review

White Paper. NexentaConnect TM Technology Review White Paper NexentaConnect TM Technology Review By Alex Aizman and Murat Karslioglu, February 2014 Table of Contents Background... 3 Storage Deployment Options, vmotion and DRS... 4 Local Storage... 5

More information

GPFS Storage Server. Concepts and Setup in Lemanicus BG/Q system" Christian Clémençon (EPFL-DIT)" " 4 April 2013"

GPFS Storage Server. Concepts and Setup in Lemanicus BG/Q system Christian Clémençon (EPFL-DIT)  4 April 2013 GPFS Storage Server Concepts and Setup in Lemanicus BG/Q system" Christian Clémençon (EPFL-DIT)" " Agenda" GPFS Overview" Classical versus GSS I/O Solution" GPFS Storage Server (GSS)" GPFS Native RAID

More information

Netapp @ 10th TF-Storage Meeting

Netapp @ 10th TF-Storage Meeting Netapp @ 10th TF-Storage Meeting Wojciech Janusz, Netapp Poland Bogusz Błaszkiewicz, Netapp Poland Ljubljana, 2012.02.20 Agenda Data Ontap Cluster-Mode pnfs E-Series NetApp Confidential - Internal Use

More information

File System & Device Drive. Overview of Mass Storage Structure. Moving head Disk Mechanism. HDD Pictures 11/13/2014. CS341: Operating System

File System & Device Drive. Overview of Mass Storage Structure. Moving head Disk Mechanism. HDD Pictures 11/13/2014. CS341: Operating System CS341: Operating System Lect 36: 1 st Nov 2014 Dr. A. Sahu Dept of Comp. Sc. & Engg. Indian Institute of Technology Guwahati File System & Device Drive Mass Storage Disk Structure Disk Arm Scheduling RAID

More information

Nexenta Performance Scaling for Speed and Cost

Nexenta Performance Scaling for Speed and Cost Nexenta Performance Scaling for Speed and Cost Key Features Optimize Performance Optimize Performance NexentaStor improves performance for all workloads by adopting commodity components and leveraging

More information

Introduction to Gluster. Versions 3.0.x

Introduction to Gluster. Versions 3.0.x Introduction to Gluster Versions 3.0.x Table of Contents Table of Contents... 2 Overview... 3 Gluster File System... 3 Gluster Storage Platform... 3 No metadata with the Elastic Hash Algorithm... 4 A Gluster

More information

Linux Powered Storage:

Linux Powered Storage: Linux Powered Storage: Building a Storage Server with Linux Architect & Senior Manager rwheeler@redhat.com June 6, 2012 1 Linux Based Systems are Everywhere Used as the base for commercial appliances Enterprise

More information

Philip Lawrence. Senior Solution Architect Sun Microsystems UK & Ireland

Philip Lawrence. Senior Solution Architect Sun Microsystems UK & Ireland > Philip Lawrence Senior Solution Architect Sun Microsystems UK & Ireland 1 Disruptive Innovation Term coined by Clayton Christensen in his book The Innovator's Dilemma to denote a technology that aside

More information

Solid State Storage in the Evolution of the Data Center

Solid State Storage in the Evolution of the Data Center Solid State Storage in the Evolution of the Data Center Trends and Opportunities Bruce Moxon CTO, Systems and Solutions stec Presented at the Lazard Capital Markets Solid State Storage Day New York, June

More information

ntier Verde Simply Affordable File Storage

ntier Verde Simply Affordable File Storage ntier Verde Simply Affordable File Storage Current Market Problems Data Growth Continues Data Retention Increases By 2020 the Digital Universe will hold 40 Zettabytes The Market is Missing: An easy to

More information

DIABLO TECHNOLOGIES MEMORY CHANNEL STORAGE AND VMWARE VIRTUAL SAN : VDI ACCELERATION

DIABLO TECHNOLOGIES MEMORY CHANNEL STORAGE AND VMWARE VIRTUAL SAN : VDI ACCELERATION DIABLO TECHNOLOGIES MEMORY CHANNEL STORAGE AND VMWARE VIRTUAL SAN : VDI ACCELERATION A DIABLO WHITE PAPER AUGUST 2014 Ricky Trigalo Director of Business Development Virtualization, Diablo Technologies

More information

An Oracle White Paper August 2011. Deploying Hybrid Storage Pools with Oracle Flash Technology and the Oracle Solaris ZFS File System

An Oracle White Paper August 2011. Deploying Hybrid Storage Pools with Oracle Flash Technology and the Oracle Solaris ZFS File System An Oracle White Paper August 2011 Deploying Hybrid Storage Pools with Oracle Flash Technology and the Oracle Solaris ZFS File System Introduction... 1 Flash Technology Moves to the Enterprise... 3 Enterprise

More information

WHITE PAPER. Software Defined Storage Hydrates the Cloud

WHITE PAPER. Software Defined Storage Hydrates the Cloud WHITE PAPER Software Defined Storage Hydrates the Cloud Table of Contents Overview... 2 NexentaStor (Block & File Storage)... 4 Software Defined Data Centers (SDDC)... 5 OpenStack... 5 CloudStack... 6

More information

Distributed File System Choices: Red Hat Storage, GFS2 & pnfs

Distributed File System Choices: Red Hat Storage, GFS2 & pnfs Distributed File System Choices: Red Hat Storage, GFS2 & pnfs Ric Wheeler Architect & Senior Manager, Red Hat June 27, 2012 Overview Distributed file system basics Red Hat distributed file systems Performance

More information

The Panasas Parallel Storage Cluster. Acknowledgement: Some of the material presented is under copyright by Panasas Inc.

The Panasas Parallel Storage Cluster. Acknowledgement: Some of the material presented is under copyright by Panasas Inc. The Panasas Parallel Storage Cluster What Is It? What Is The Panasas ActiveScale Storage Cluster A complete hardware and software storage solution Implements An Asynchronous, Parallel, Object-based, POSIX

More information

Commoditisation of the High-End Research Storage Market with the Dell MD3460 & Intel Enterprise Edition Lustre

Commoditisation of the High-End Research Storage Market with the Dell MD3460 & Intel Enterprise Edition Lustre Commoditisation of the High-End Research Storage Market with the Dell MD3460 & Intel Enterprise Edition Lustre University of Cambridge, UIS, HPC Service Authors: Wojciech Turek, Paul Calleja, John Taylor

More information

Storage Architectures for Big Data in the Cloud

Storage Architectures for Big Data in the Cloud Storage Architectures for Big Data in the Cloud Sam Fineberg HP Storage CT Office/ May 2013 Overview Introduction What is big data? Big Data I/O Hadoop/HDFS SAN Distributed FS Cloud Summary Research Areas

More information

Dial up or down the flash in the system depending on requirements Same OS, feature set or uxp whether you;re using hybrid or AFA

Dial up or down the flash in the system depending on requirements Same OS, feature set or uxp whether you;re using hybrid or AFA SFD6 - Wednesday PM Tegile *Narayan (CMO) Business transformation through storage transformation First product shipped Feb 2011 Close to 700 customers Focused on enterprise mid-range 1200 systems in the

More information

Large Scale Storage. Orlando Richards, Information Services orlando.richards@ed.ac.uk. LCFG Users Day, University of Edinburgh 18 th January 2013

Large Scale Storage. Orlando Richards, Information Services orlando.richards@ed.ac.uk. LCFG Users Day, University of Edinburgh 18 th January 2013 Large Scale Storage Orlando Richards, Information Services orlando.richards@ed.ac.uk LCFG Users Day, University of Edinburgh 18 th January 2013 Overview My history of storage services What is (and is not)

More information

CloudByte ElastiStor Date: February 2014 Author: Tony Palmer, Senior Lab Analyst

CloudByte ElastiStor Date: February 2014 Author: Tony Palmer, Senior Lab Analyst ESG Lab Review CloudByte ElastiStor Date: February 2014 Author: Tony Palmer, Senior Lab Analyst Abstract: This ESG Lab review documents the hands-on evaluation of CloudByte ElastiStor unified storage on

More information

SLIDE 1 www.bitmicro.com. Previous Next Exit

SLIDE 1 www.bitmicro.com. Previous Next Exit SLIDE 1 MAXio All Flash Storage Array Popular Applications MAXio N1A6 SLIDE 2 MAXio All Flash Storage Array Use Cases High speed centralized storage for IO intensive applications email, OLTP, databases

More information

The IntelliMagic White Paper: Storage Performance Analysis for an IBM Storwize V7000

The IntelliMagic White Paper: Storage Performance Analysis for an IBM Storwize V7000 The IntelliMagic White Paper: Storage Performance Analysis for an IBM Storwize V7000 Summary: This document describes how to analyze performance on an IBM Storwize V7000. IntelliMagic 2012 Page 1 This

More information

89 Fifth Avenue, 7th Floor. New York, NY 10003. www.theedison.com 212.367.7400. White Paper. HP 3PAR Adaptive Flash Cache: A Competitive Comparison

89 Fifth Avenue, 7th Floor. New York, NY 10003. www.theedison.com 212.367.7400. White Paper. HP 3PAR Adaptive Flash Cache: A Competitive Comparison 89 Fifth Avenue, 7th Floor New York, NY 10003 www.theedison.com 212.367.7400 White Paper HP 3PAR Adaptive Flash Cache: A Competitive Comparison Printed in the United States of America Copyright 2014 Edison

More information

Direct NFS - Design considerations for next-gen NAS appliances optimized for database workloads Akshay Shah Gurmeet Goindi Oracle

Direct NFS - Design considerations for next-gen NAS appliances optimized for database workloads Akshay Shah Gurmeet Goindi Oracle Direct NFS - Design considerations for next-gen NAS appliances optimized for database workloads Akshay Shah Gurmeet Goindi Oracle Agenda Introduction Database Architecture Direct NFS Client NFS Server

More information

HPC data becomes Big Data. Peter Braam peter.braam@braamresearch.com

HPC data becomes Big Data. Peter Braam peter.braam@braamresearch.com HPC data becomes Big Data Peter Braam peter.braam@braamresearch.com me 1983-2000 Academia Maths & Computer Science Entrepreneur with startups (5x) 4 startups sold Lustre emerged Held executive jobs with

More information

Panasas at the RCF. Fall 2005 Robert Petkus RHIC/USATLAS Computing Facility Brookhaven National Laboratory. Robert Petkus Panasas at the RCF

Panasas at the RCF. Fall 2005 Robert Petkus RHIC/USATLAS Computing Facility Brookhaven National Laboratory. Robert Petkus Panasas at the RCF Panasas at the RCF HEPiX at SLAC Fall 2005 Robert Petkus RHIC/USATLAS Computing Facility Brookhaven National Laboratory Centralized File Service Single, facility-wide namespace for files. Uniform, facility-wide

More information

<Insert Picture Here> Btrfs Filesystem

<Insert Picture Here> Btrfs Filesystem Btrfs Filesystem Chris Mason Btrfs Goals General purpose filesystem that scales to very large storage Feature focused, providing features other Linux filesystems cannot Administration

More information

Scientific Storage at FNAL. Gerard Bernabeu Altayo Dmitry Litvintsev Gene Oleynik 14/10/2015

Scientific Storage at FNAL. Gerard Bernabeu Altayo Dmitry Litvintsev Gene Oleynik 14/10/2015 Scientific Storage at FNAL Gerard Bernabeu Altayo Dmitry Litvintsev Gene Oleynik 14/10/2015 Index - Storage use cases - Bluearc - Lustre - EOS - dcache disk only - dcache+enstore Data distribution by solution

More information

An Oracle Technical White Paper December 2011. Reference Architecture For VistaInsight for Networks on Oracle Server And Storage

An Oracle Technical White Paper December 2011. Reference Architecture For VistaInsight for Networks on Oracle Server And Storage An Oracle Technical White Paper December 2011 Reference Architecture For VistaInsight for Networks on Oracle Server And Storage Overview... 2 Application Overview... 2 About InfoVista... 2 Vista Insight

More information

BlueArc unified network storage systems 7th TF-Storage Meeting. Scale Bigger, Store Smarter, Accelerate Everything

BlueArc unified network storage systems 7th TF-Storage Meeting. Scale Bigger, Store Smarter, Accelerate Everything BlueArc unified network storage systems 7th TF-Storage Meeting Scale Bigger, Store Smarter, Accelerate Everything BlueArc s Heritage Private Company, founded in 1998 Headquarters in San Jose, CA Highest

More information

CSE-E5430 Scalable Cloud Computing P Lecture 5

CSE-E5430 Scalable Cloud Computing P Lecture 5 CSE-E5430 Scalable Cloud Computing P Lecture 5 Keijo Heljanko Department of Computer Science School of Science Aalto University keijo.heljanko@aalto.fi 12.10-2015 1/34 Fault Tolerance Strategies for Storage

More information

The Data Placement Challenge

The Data Placement Challenge The Data Placement Challenge Entire Dataset Applications Active Data Lowest $/IOP Highest throughput Lowest latency 10-20% Right Place Right Cost Right Time 100% 2 2 What s Driving the AST Discussion?

More information

SUN STORAGE 7000 UNIFIED STORAGE SYSTEMS

SUN STORAGE 7000 UNIFIED STORAGE SYSTEMS > SUN STORAGE 7000 UNIFIED STORAGE SYSTEMS IT S TIME TO CHANGE YOUR STORAGE Presenter s Name Presenter s Title Presenter s Company 1 Sun Storage 7000 Unified Storage Systems Agenda The Sun Storage 7000

More information

Architecting a High Performance Storage System

Architecting a High Performance Storage System WHITE PAPER Intel Enterprise Edition for Lustre* Software High Performance Data Division Architecting a High Performance Storage System January 2014 Contents Introduction... 1 A Systematic Approach to

More information

Maximize your Engineered Systems

Maximize your Engineered Systems Maximize your Engineered Systems Jan Poos Principal Sales Consultant EMEA Disk BU @JanStorage Safe Harbor Statement The following is intended to outline our general product direction. It is intended for

More information

IBM System x GPFS Storage Server

IBM System x GPFS Storage Server IBM System x GPFS Storage Server Schöne Aussicht en für HPC Speicher ZKI-Arbeitskreis Paderborn, 15.03.2013 Karsten Kutzer Client Technical Architect Technical Computing IBM Systems & Technology Group

More information

Flash Storage: Trust, But Verify

Flash Storage: Trust, But Verify Flash Storage: Trust, But Verify A better, vendor-independent way to analyze flash performance Leah Schoeb, Evaluator Group Peter Murray, Load DynamiX 1 Introductions 2 Speakers Leah Schoeb Senior Partner

More information

www.thinkparq.com www.beegfs.com

www.thinkparq.com www.beegfs.com www.thinkparq.com www.beegfs.com KEY ASPECTS Maximum Flexibility Maximum Scalability BeeGFS supports a wide range of Linux distributions such as RHEL/Fedora, SLES/OpenSuse or Debian/Ubuntu as well as a

More information

Using SmartOS as a Hypervisor

Using SmartOS as a Hypervisor Using SmartOS as a Hypervisor SCALE 10x Robert Mustacchi rm@joyent.com (@rmustacc) Software Engineer What is SmartOS? Solaris heritage Zones - OS level virtualization Crossbow - virtual NICs ZFS - pooled

More information

MaxDeploy Hyper- Converged Reference Architecture Solution Brief

MaxDeploy Hyper- Converged Reference Architecture Solution Brief MaxDeploy Hyper- Converged Reference Architecture Solution Brief MaxDeploy Reference Architecture solutions are configured and tested for support with Maxta software- defined storage and with industry

More information

Measuring Interface Latencies for SAS, Fibre Channel and iscsi

Measuring Interface Latencies for SAS, Fibre Channel and iscsi Measuring Interface Latencies for SAS, Fibre Channel and iscsi Dennis Martin Demartek President Santa Clara, CA 1 Demartek Company Overview Industry analysis with on-site test lab Lab includes servers,

More information

ZFS The Last Word in File Systems. Jeff Bonwick Bill Moore www.opensolaris.org/os/community/zfs

ZFS The Last Word in File Systems. Jeff Bonwick Bill Moore www.opensolaris.org/os/community/zfs ZFS The Last Word In File Systems Jeff Bonwick Bill Moore www.opensolaris.org/os/community/zfs Page 1 ZFS Overview Pooled storage Transactional object system Completely eliminates the antique notion of

More information

High Availability Databases based on Oracle 10g RAC on Linux

High Availability Databases based on Oracle 10g RAC on Linux High Availability Databases based on Oracle 10g RAC on Linux WLCG Tier2 Tutorials, CERN, June 2006 Luca Canali, CERN IT Outline Goals Architecture of an HA DB Service Deployment at the CERN Physics Database

More information

UNIFIED HYBRID STORAGE. Performance, Availability and Scale for Any SAN and NAS Workload in Your Environment

UNIFIED HYBRID STORAGE. Performance, Availability and Scale for Any SAN and NAS Workload in Your Environment DATASHEET TM NST6000 UNIFIED HYBRID STORAGE Performance, Availability and Scale for Any SAN and NAS Workload in Your Environment UNIFIED The Nexsan NST6000 unified hybrid storage appliance is ideal for

More information

IOmark- VDI. Nimbus Data Gemini Test Report: VDI- 130906- a Test Report Date: 6, September 2013. www.iomark.org

IOmark- VDI. Nimbus Data Gemini Test Report: VDI- 130906- a Test Report Date: 6, September 2013. www.iomark.org IOmark- VDI Nimbus Data Gemini Test Report: VDI- 130906- a Test Copyright 2010-2013 Evaluator Group, Inc. All rights reserved. IOmark- VDI, IOmark- VDI, VDI- IOmark, and IOmark are trademarks of Evaluator

More information

Storage Management for the Oracle Database on Red Hat Enterprise Linux 6: Using ASM With or Without ASMLib

Storage Management for the Oracle Database on Red Hat Enterprise Linux 6: Using ASM With or Without ASMLib Storage Management for the Oracle Database on Red Hat Enterprise Linux 6: Using ASM With or Without ASMLib Sayan Saha, Sue Denham & Lars Herrmann 05/02/2011 On 22 March 2011, Oracle posted the following

More information

VDI Optimization Real World Learnings. Russ Fellows, Evaluator Group

VDI Optimization Real World Learnings. Russ Fellows, Evaluator Group Russ Fellows, Evaluator Group SNIA Legal Notice The material contained in this tutorial is copyrighted by the SNIA unless otherwise noted. Member companies and individual members may use this material

More information

Lab Evaluation of NetApp Hybrid Array with Flash Pool Technology

Lab Evaluation of NetApp Hybrid Array with Flash Pool Technology Lab Evaluation of NetApp Hybrid Array with Flash Pool Technology Evaluation report prepared under contract with NetApp Introduction As flash storage options proliferate and become accepted in the enterprise,

More information

Investigation of storage options for scientific computing on Grid and Cloud facilities

Investigation of storage options for scientific computing on Grid and Cloud facilities Investigation of storage options for scientific computing on Grid and Cloud facilities Overview Context Test Bed Lustre Evaluation Standard benchmarks Application-based benchmark HEPiX Storage Group report

More information

THE SUMMARY. ARKSERIES - pg. 3. ULTRASERIES - pg. 5. EXTREMESERIES - pg. 9

THE SUMMARY. ARKSERIES - pg. 3. ULTRASERIES - pg. 5. EXTREMESERIES - pg. 9 PRODUCT CATALOG THE SUMMARY ARKSERIES - pg. 3 ULTRASERIES - pg. 5 EXTREMESERIES - pg. 9 ARK SERIES THE HIGH DENSITY STORAGE FOR ARCHIVE AND BACKUP Unlimited scalability Painless Disaster Recovery The ARK

More information

June 2009. Blade.org 2009 ALL RIGHTS RESERVED

June 2009. Blade.org 2009 ALL RIGHTS RESERVED Contributions for this vendor neutral technology paper have been provided by Blade.org members including NetApp, BLADE Network Technologies, and Double-Take Software. June 2009 Blade.org 2009 ALL RIGHTS

More information

HPC @ CRIBI. Calcolo Scientifico e Bioinformatica oggi Università di Padova 13 gennaio 2012

HPC @ CRIBI. Calcolo Scientifico e Bioinformatica oggi Università di Padova 13 gennaio 2012 HPC @ CRIBI Calcolo Scientifico e Bioinformatica oggi Università di Padova 13 gennaio 2012 what is exact? experience on advanced computational technologies a company lead by IT experts with a strong background

More information

Ceph. A file system a little bit different. Udo Seidel

Ceph. A file system a little bit different. Udo Seidel Ceph A file system a little bit different Udo Seidel Ceph what? So-called parallel distributed cluster file system Started as part of PhD studies at UCSC Public announcement in 2006 at 7 th OSDI File system

More information

VMware VSAN och Virtual Volumer

VMware VSAN och Virtual Volumer VMware VSAN och Virtual Volumer Claes Bäckström RTS Här läggs leverantörsloggor om det finns VMware VSAN 6.0 Här läggs leverantörsloggor om det finns VMware Virtual SAN 6.0 Radically Simple Hypervisor-Converged

More information

NetApp High-Performance Computing Solution for Lustre: Solution Guide

NetApp High-Performance Computing Solution for Lustre: Solution Guide Technical Report NetApp High-Performance Computing Solution for Lustre: Solution Guide Robert Lai, NetApp August 2012 TR-3997 TABLE OF CONTENTS 1 Introduction... 5 1.1 NetApp HPC Solution for Lustre Introduction...5

More information

Maurice Askinazi Ofer Rind Tony Wong. HEPIX @ Cornell Nov. 2, 2010 Storage at BNL

Maurice Askinazi Ofer Rind Tony Wong. HEPIX @ Cornell Nov. 2, 2010 Storage at BNL Maurice Askinazi Ofer Rind Tony Wong HEPIX @ Cornell Nov. 2, 2010 Storage at BNL Traditional Storage Dedicated compute nodes and NFS SAN storage Simple and effective, but SAN storage became very expensive

More information

Choosing Storage Systems

Choosing Storage Systems Choosing Storage Systems For MySQL Peter Zaitsev, CEO Percona Percona Live MySQL Conference and Expo 2013 Santa Clara,CA April 25,2013 Why Right Choice for Storage is Important? 2 because Wrong Choice

More information

Preview of a Novel Architecture for Large Scale Storage

Preview of a Novel Architecture for Large Scale Storage Preview of a Novel Architecture for Large Scale Storage Andreas Petzold, Christoph-Erdmann Pfeiler, Jos van Wezel Steinbuch Centre for Computing STEINBUCH CENTRE FOR COMPUTING - SCC KIT University of the

More information

Lessons learned from parallel file system operation

Lessons learned from parallel file system operation Lessons learned from parallel file system operation Roland Laifer STEINBUCH CENTRE FOR COMPUTING - SCC KIT University of the State of Baden-Württemberg and National Laboratory of the Helmholtz Association

More information

HADOOP ON ORACLE ZFS STORAGE A TECHNICAL OVERVIEW

HADOOP ON ORACLE ZFS STORAGE A TECHNICAL OVERVIEW HADOOP ON ORACLE ZFS STORAGE A TECHNICAL OVERVIEW 757 Maleta Lane, Suite 201 Castle Rock, CO 80108 Brett Weninger, Managing Director brett.weninger@adurant.com Dave Smelker, Managing Principal dave.smelker@adurant.com

More information

EMC XTREMIO EXECUTIVE OVERVIEW

EMC XTREMIO EXECUTIVE OVERVIEW EMC XTREMIO EXECUTIVE OVERVIEW COMPANY BACKGROUND XtremIO develops enterprise data storage systems based completely on random access media such as flash solid-state drives (SSDs). By leveraging the underlying

More information

The safer, easier way to help you pass any IT exams. Exam : 000-115. Storage Sales V2. Title : Version : Demo 1 / 5

The safer, easier way to help you pass any IT exams. Exam : 000-115. Storage Sales V2. Title : Version : Demo 1 / 5 Exam : 000-115 Title : Storage Sales V2 Version : Demo 1 / 5 1.The IBM TS7680 ProtecTIER Deduplication Gateway for System z solution is designed to provide all of the following EXCEPT: A. ESCON attach

More information

ARCHITECTURE WHITE PAPER ARCHITECTURE WHITE PAPER

ARCHITECTURE WHITE PAPER ARCHITECTURE WHITE PAPER ARCHITECTURE WHITE PAPER ARCHITECTURE WHITE PAPER Table of Contents Data Value... 2 NexGen Storage Architecture... 3 Storage Quality of Service and Service Levels... 4 PCIe Flash-first Design... 9 Dynamic

More information

General Parallel File System (GPFS) Native RAID For 100,000-Disk Petascale Systems

General Parallel File System (GPFS) Native RAID For 100,000-Disk Petascale Systems General Parallel File System (GPFS) Native RAID For 100,000-Disk Petascale Systems Veera Deenadhayalan IBM Almaden Research Center 2011 IBM Corporation Hard Disk Rates Are Lagging There have been recent

More information

Lustre * Filesystem for Cloud and Hadoop *

Lustre * Filesystem for Cloud and Hadoop * OpenFabrics Software User Group Workshop Lustre * Filesystem for Cloud and Hadoop * Robert Read, Intel Lustre * for Cloud and Hadoop * Brief Lustre History and Overview Using Lustre with Hadoop Intel Cloud

More information

IBM ELASTIC STORAGE SEAN LEE

IBM ELASTIC STORAGE SEAN LEE IBM ELASTIC STORAGE SEAN LEE Solution Architect Platform Computing Division IBM Greater China Group Agenda Challenges in Data Management What is IBM Elastic Storage Key Features Elastic Storage Server

More information

FAS6200 Cluster Delivers Exceptional Block I/O Performance with Low Latency

FAS6200 Cluster Delivers Exceptional Block I/O Performance with Low Latency FAS6200 Cluster Delivers Exceptional Block I/O Performance with Low Latency Dimitris Krekoukias Systems Engineer NetApp Data ONTAP 8 software operating in Cluster-Mode is the industry's only unified, scale-out

More information

THE EMC ISILON STORY. Big Data In The Enterprise. Copyright 2012 EMC Corporation. All rights reserved.

THE EMC ISILON STORY. Big Data In The Enterprise. Copyright 2012 EMC Corporation. All rights reserved. THE EMC ISILON STORY Big Data In The Enterprise 2012 1 Big Data In The Enterprise Isilon Overview Isilon Technology Summary 2 What is Big Data? 3 The Big Data Challenge File Shares 90 and Archives 80 Bioinformatics

More information

Springpath Data Platform

Springpath Data Platform Springpath Data Platform WHITE PAPER February 2015 SUMMARY SPRINGPATH DATA PLATFORM Introduction 1 HALO Architecture 3 Data Distribution 3 Data Caching and Data Persistence 4 Data Optimization 5 Data Services

More information

New Features in PSP2 for SANsymphony -V10 Software-defined Storage Platform and DataCore Virtual SAN

New Features in PSP2 for SANsymphony -V10 Software-defined Storage Platform and DataCore Virtual SAN New Features in PSP2 for SANsymphony -V10 Software-defined Storage Platform and DataCore Virtual SAN Updated: May 19, 2015 Contents Introduction... 1 Cloud Integration... 1 OpenStack Support... 1 Expanded

More information

Using Nexenta as a cloud storage

Using Nexenta as a cloud storage University of Ljubljana Faculty of Computer and Information Science Using Nexenta as a cloud storage dr. Matjaž Pančur, UL FRI dr. Mojca Ciglarič, UL FRI Agenda Standardization and commoditization of storage

More information