Algorithms and Methods for Distributed Storage Networks 8 Storage Virtualization and DHT Christian Schindelhauer

Size: px
Start display at page:

Download "Algorithms and Methods for Distributed Storage Networks 8 Storage Virtualization and DHT Christian Schindelhauer"

Transcription

1 Algorithms and Methods for Distributed Storage Networks 8 Storage Virtualization and DHT Institut für Informatik Wintersemester 2007/08

2 Overview Concept of Virtualization Storage Area Networks Principles Optimization Distributed File Systems Without virtualization, e.g. Network File Systems With virtualization, e.g. Google File System Distributed Wide Area Storage Networks Distributed Hash Tables Peer-to-Peer Storage 2

3 Concept of Virtualization Principle A virtual storage constitutes handles all application accesses to the file system The virtual disk partitions files and stores blocks over several (physical) hard disks Control mechanisms allow redundancy and failure repair Control Virtualization server assigns data, e.g. blocks of files to hard disks (address space remapping) Controls replication and redundancy strategy Adds and removes storage devices 3 File Hard Disks Virtual Disk

4 Storage Virtualization Capabilities Replication Pooling Disk Management Advantages Data migration Higher availability Simple maintenance Scalability Disadvantages Un-installing is time consuming Compatibility and interoperability Complexity of the system Classic Implementation Host-based - Logical Volume Management - File Systems, e.g. NFS Storage devices based - RAID Network based - Storage Area Network New approaches Distributed Wide Area Storage Networks Distributed Hash Tables Peer-to-Peer Storage 4

5 Storage Area Networks Virtual Block Devices without file system connects hard disks Advantages simpler storage administration more flexible servers can boot from the SAN effective disaster recovery allows storage replication Compatibility problems between hard disks and virtualization server 5

6 SAN Networking Networking FCP (Fibre Channel Protocol) - SCSI over Fibre Channel iscsi (SCSI over TCP/IP) HyperSCSI (SCSI over Ethernet) ATA over Ethernet Fibre Channel over Ethernet iscsi over InfiniBand FCP over IP 6

7 SAN File Systems File system for concurrent read and write operations by multiple computers without conventional file locking concurrent direct access to blocks by servers Examples Veritas Cluster File System Xsan Global File System Oracle Cluster File System VMware VMFS IBM General Parallel File System 7

8 Distributed File Systems (without Virtualization) aka. Network File System Supports sharing of files, tapes, printers etc. Allows multiple client processes on multiple hosts to read and write the same files concurrency control or locking mechanisms necessary Examples Network File System (NFS) Server Message Block (SMB), Samba Apple Filing Protocol (AFP) Amazon Simple Storage Service (S3) 8

9 Distributed File Systems with Virtualization Example: Google File System File system on top of other file systems with builtin virtualization System built from cheap standard components (with high failure rates) Few large files Only operations: read, create, append, delete - concurrent appends and reads must be handled High bandwidth important Replication strategy chunk replication master replication Application GFS client (file name, chunk index) (chunk handle, chunk locations) (chunk handle, byte range) chunk data GFS master File namespace /foo/bar chunk 2ef0 Instructions to chunkserver GFS chunkserver Linux file system Chunkserver state GFS chunkserver Linux file system Figure 1: GFS Architecture 4 step 1 file region may end up contain Client Master clients, although the replicas wi and replication decisions using global knowledge. However, 2 3 tent TCP connection to the chunkserver dividual over anoperations extended are comple we must minimize its involvement in reads and writes so period of time. Third, it reduces the size order of on theall metadata replicas. This leave that it does not become a bottleneck. Clients never read stored on the master. This allows us tobut keep undefined the metadata state as noted in and write file data through the master. Instead, asecondary client asks in memory, which in turn brings other advantages that we the master which chunkservers it should contact. Replica It caches A will discuss in Section this information for a limited time and interacts with the On the other hand, a large chunk size, 3.2 even with Data lazy space Flow chunkservers directly for many subsequent operations. allocation, has its disadvantages. A small We file decouple consists of the a flow of data Let us explain the interactions for a simple 7read with reference to Figure 1. First, using the fixed chunk size, the client storing 5 those chunks may become hot spots if many clients small number of chunks, perhaps just one. The chunkservers Primary use the network efficiently. W translates the file name and byte offset specified Replica by the application into a chunk index within the file. Then, it sends been a major Legend: issue because our applications pushedmostly linearlyread along a carefully are accessing the same file. In practice, client hot spots to the have primary not and then the master a request containing the file name and chunk large multi-chunk files sequentially. in a pipelined fashion. Our go index. The master replies with the corresponding chunk However, hot spots Control did develop whenmachine s GFS was first network used bandwidth, 6 handle and locations of the replicas. The client Secondary caches this by a batch-queue system: Data an executableand was written high-latency to GFSlinks, and m information using the file name and chunk index Replica as the key. B as a single-chunk file and then started through on hundreds all the of machines at the same time. The few chunkservers To fully storing utilize thiseach machin data. The client then sends a request to one of the replicas, most likely the closest one. The request specifies the chunk executable were overloaded by hundredsdata of simultaneous is pushed linearly requests. and We Data fixed this Flow problem by storing such executables along a handle and a byte range within that chunk. Figure Further 2: Write reads Control than distributed in some other of the same chunk require no more client-master interaction with a higher replication factor and by making the batchqueue system stagger application start times. A potential each machine s full outbound b until the cached information expires or the file is reopened. The Google File System In fact, the client typically asks for multiple chunks in the long-term solution is to allow clients to read fer the data data fromas other fast as possible Sanjay Ghemawat, same Howard requestgobioff, and the master and Shun-Tak can also becomes include Leung the unreachable information for chunks immediately following those or clients repliesinthat suchit situations. no longer holds multiple recipients. a lease. requested. This To avoid network bottlenecks extra information 9 sidesteps several future client-master inter-switch links are often bot 3. The client pushes Albert-Ludwigs-Universität interactions at practically no extra cost. The master stores three major types of machine metadata: forwards the filethe data to t 2.6 Metadata the data to all the replicas. A clientfreiburg can do so in any order. and Each chunk Christian chunkserver namespaces, will Schindelhauer thestore mapping from network files topology chunks, that has no 2.5 Chunk Size the data in an internaland LRU thebuffer locations cache of each until chunk s the replicas. client Allis metadata pushingisdata to chun Chunk size is one of the key design parameters. data is usedwe orhave aged out. kept By decoupling the master s thememory. data flowthe first sends two types the data (names- can and improve file-to-chunk performance mapping) by are also kept persistent by to the closest c chosen 64 MB, which is much larger than from typical the control file system block sizes. Each chunk replicascheduling is stored asthe a plain expensivelogging data flow mutations basedtonan the operation net- log stored on the mas- flow, wepaces wards it to the closest chunkser Linux file on a chunkserver and is extended S1, say S2. Similarly, S2 forwar work only topology as needed. regardless ter s of which local disk chunkserver and replicated is the on remote machines. Using Lazy space allocation avoids wasting space is closer to S2, and so on. Our primary. due to Section internal 3.2 discusses a log allows this further. us to update the master state simply, reliably, Legend: Data messages Control messages

10 Distributed Wide Area Storage Networks Distributed Hash Tables Relieving hot spots in the Internet Caching strategies for web servers Peer-to-Peer Networks Distributed file lookup and download in Overlay networks Most (or the best) of them use: DHT 10

11 WWW Load Balancing Web surfing: Web servers offer web pages Web clients request web pages Most of the time these requests are independent Requests use resources of the web servers bandwidth computation time Christian Stefan Arne 11

12 Load Some web servers have always high load for permanent high loads servers must be sufficiently powerful Some suffer under high fluctuations e.g. special events: - jpl.nasa.gov (Mars mission) - cnn.com (terrorist attack) Monday Tuesday Wednesday Server extension for worst case not reasonable Serving the requests is desired 12

13 Load Balancing in the WWW Monday Tuesday Wednesday Fluctuations target some servers A B A B A B (Commercial) solution Service providers offer exchange servers an Many requests will be distributed among these servers But how? A B 13

14 Literature Leighton, Lewin, et al. STOC 97 Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web Used by Akamai (founded 1997) Web-Cache 14

15 Start Situation Without load balancing Advantage simple Disadvantage servers must be designed for worst case situations Web-Server Web pages request Web-Clients 15

16 Site Caching Web-Server The whole web-site is copied to different web caches Browsers request at web server Web server redirects requests to Web- Cache Web-Cache delivers Web pages Advantage: good load balancing Disadvantage: bottleneck: redirect large overhead for complete web-site replication redirect Web-Cache Web-Clients 16

17 Proxy Caching Web-Server Each web page is distributed to a few web-caches Only first request is sent to web server Links reference to pages in the webcache Advantage: No bottleneck Disadvantages: Load balancing only implicit High requirements for placements redirect Link request Then, web clients surfs in the webcache Web- Cache Web-Client 17

18 Requirements Balance fair balancing of web pages Dynamics Efficient insert and delete of webcache-servers and files?? new X X Views Web-Clients see different set of web-caches 18

19 Hash Functions Buckets Items Set of Items: Set of Buckets: Example: 19

20 Ranged Hash-Funktionen Given: Items, Number Caches (Buckets), Bucket set: Views Ranged Hash-Funktion: Prerequisite: for alle views Buckets View Items 20

21 First Idea: Hash Function Algorithm: Choose Hash funktion, e.g. 3 i + 1 mod Balance: n: number of Cache servers very good Dynamics Insert or remove of a single cache server New hash functions and total rehashing Very expensive!! i + 2 mod X

22 Requirements of the Ranged Hash Functions Monotony After adding or removing new caches (buckets) no pages (items) should be moved Balance All caches should have the same load Spread (Verbreitung,Streuung) A page should be distributed to a bounded number of Load caches No Cache should not have substantially more load than the average 22

23 Monotony After adding or removing new caches (buckets) no pages (items) should be moved Formally: For all Pages View 1: View 2: Caches Caches Pages 23

24 Balance For every view V the is the f V (i) balanced For a constant c and all : Pages View 1: View 2: Caches Caches Pages 24

25 Spread The spread (i) of a page i is the overall number of all necessary copies (over all views) View 1: View 2: View 3: 25

26 Load The load λ(b) of a cache b is the over-all number of all copies (over all views) wher in View V := set of all pages assigned to bucket b View 1: View 2: λ(b 1 ) = 2 λ(b 2 ) = 3 View 3: b 1 b 2 26

27 Distributed Hash Tables Theorem There exists a family of hash function with the following properties Each function f F is monotone Balance: For every view C number of caches (Buckets) C/t minimum number of caches per View V/C = constant (#Views / #Caches) I = C (# pages = # Caches) Spread: For each page i with probability Load: For each cache b mit W keit 27

28 The Design 2 Hash functions onto the reals [0,1] maps k log C copies of cache b randomly to [0,1] maps web page i randomly to the interval [0,1] := Cache, which minimizes Caches (Buckets): View View Webseiten (Items): 28

29 := Cache which minimizes For all : Monotony Observe: blue interval in V 2 and in V 1 empty! View View

30 Balance: For all views 2. Balance Caches (Buckets): Choose fixed view and a web page i Apply hash functions and. Under the assumption that the mapping is random every cache is chosen with the same probability View 0 1 Webseiten (Items): 30

31 3. Spread (i) = number of all necessary copies (over all views) C number of caches (Buckets) C/t minimum number of caches per View V/C = constant (#Views / #Caches) I = C (# pages = # Caches) ever user knows at least a fraction of 1/t over the caches For every page i with prob. Proof sketch: Every view has a cache in an interval of length t/c (with high probability) The number of caches gives an upper bound for the spread 0 t/c 2t/C 1 31

32 4. Load Last (load): λ(b) = Number of copies over all views where := wet of pages assigned to bucket b under view V For every cache be we observer with probability Proof sketch: Consider intervals of length t/c With high probability a cache of every view falls into one of these intervals The number of items in the interval gives an upper bound for the load 0 t/c 2t/C 1 32

33 Summary Distributed Hash Table is a distributed data structure for virtualization with fair balance provides dynamic behavior Standard data structure for dynamic distributed storages 33

34 Algorithms and Methods for Distributed Storage Networks 8 Storage Virtualization and DHT Institut für Informatik Wintersemester 2007/08

Distributed Storage Networks and Computer Forensics

Distributed Storage Networks and Computer Forensics Distributed Storage Networks and Computer Forensics 7 Storage Virtualization and DHT Technical Faculty Winter Semester 2011/12 Overview Concept of Virtualization Storage Area Networks Principles Optimization

More information

Christian Schindelhauer Technical Faculty Computer-Networks and Telematics University of Freiburg

Christian Schindelhauer Technical Faculty Computer-Networks and Telematics University of Freiburg DAAD Summerschool Curitiba 2011 Aspects of Large Scale High Speed Computing Building Blocks of a Cloud Storage Networks 2: Virtualization of Storage: RAID, SAN and Virtualization Christian Schindelhauer

More information

The Google File System

The Google File System The Google File System By Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung (Presented at SOSP 2003) Introduction Google search engine. Applications process lots of data. Need good file system. Solution:

More information

Distributed File System. MCSN N. Tonellotto Complements of Distributed Enabling Platforms

Distributed File System. MCSN N. Tonellotto Complements of Distributed Enabling Platforms Distributed File System 1 How do we get data to the workers? NAS Compute Nodes SAN 2 Distributed File System Don t move data to workers move workers to the data! Store data on the local disks of nodes

More information

The Google File System

The Google File System The Google File System Motivations of NFS NFS (Network File System) Allow to access files in other systems as local files Actually a network protocol (initially only one server) Simple and fast server

More information

Lecture 5: GFS & HDFS! Claudia Hauff (Web Information Systems)! ti2736b-ewi@tudelft.nl

Lecture 5: GFS & HDFS! Claudia Hauff (Web Information Systems)! ti2736b-ewi@tudelft.nl Big Data Processing, 2014/15 Lecture 5: GFS & HDFS!! Claudia Hauff (Web Information Systems)! ti2736b-ewi@tudelft.nl 1 Course content Introduction Data streams 1 & 2 The MapReduce paradigm Looking behind

More information

Distributed File Systems

Distributed File Systems Distributed File Systems Paul Krzyzanowski Rutgers University October 28, 2012 1 Introduction The classic network file systems we examined, NFS, CIFS, AFS, Coda, were designed as client-server applications.

More information

Distributed File Systems

Distributed File Systems Distributed File Systems Mauro Fruet University of Trento - Italy 2011/12/19 Mauro Fruet (UniTN) Distributed File Systems 2011/12/19 1 / 39 Outline 1 Distributed File Systems 2 The Google File System (GFS)

More information

Cloud Computing at Google. Architecture

Cloud Computing at Google. Architecture Cloud Computing at Google Google File System Web Systems and Algorithms Google Chris Brooks Department of Computer Science University of San Francisco Google has developed a layered system to handle webscale

More information

IBM Global Technology Services November 2009. Successfully implementing a private storage cloud to help reduce total cost of ownership

IBM Global Technology Services November 2009. Successfully implementing a private storage cloud to help reduce total cost of ownership IBM Global Technology Services November 2009 Successfully implementing a private storage cloud to help reduce total cost of ownership Page 2 Contents 2 Executive summary 3 What is a storage cloud? 3 A

More information

Massive Data Storage

Massive Data Storage Massive Data Storage Storage on the "Cloud" and the Google File System paper by: Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung presentation by: Joshua Michalczak COP 4810 - Topics in Computer Science

More information

August 2009. Transforming your Information Infrastructure with IBM s Storage Cloud Solution

August 2009. Transforming your Information Infrastructure with IBM s Storage Cloud Solution August 2009 Transforming your Information Infrastructure with IBM s Storage Cloud Solution Page 2 Table of Contents Executive summary... 3 Introduction... 4 A Story or three for inspiration... 6 Oops,

More information

RAID. Tiffany Yu-Han Chen. # The performance of different RAID levels # read/write/reliability (fault-tolerant)/overhead

RAID. Tiffany Yu-Han Chen. # The performance of different RAID levels # read/write/reliability (fault-tolerant)/overhead RAID # The performance of different RAID levels # read/write/reliability (fault-tolerant)/overhead Tiffany Yu-Han Chen (These slides modified from Hao-Hua Chu National Taiwan University) RAID 0 - Striping

More information

Storage Systems Autumn 2009. Chapter 6: Distributed Hash Tables and their Applications André Brinkmann

Storage Systems Autumn 2009. Chapter 6: Distributed Hash Tables and their Applications André Brinkmann Storage Systems Autumn 2009 Chapter 6: Distributed Hash Tables and their Applications André Brinkmann Scaling RAID architectures Using traditional RAID architecture does not scale Adding news disk implies

More information

Google File System. Web and scalability

Google File System. Web and scalability Google File System Web and scalability The web: - How big is the Web right now? No one knows. - Number of pages that are crawled: o 100,000 pages in 1994 o 8 million pages in 2005 - Crawlable pages might

More information

Network Attached Storage. Jinfeng Yang Oct/19/2015

Network Attached Storage. Jinfeng Yang Oct/19/2015 Network Attached Storage Jinfeng Yang Oct/19/2015 Outline Part A 1. What is the Network Attached Storage (NAS)? 2. What are the applications of NAS? 3. The benefits of NAS. 4. NAS s performance (Reliability

More information

Moving Virtual Storage to the Cloud. Guidelines for Hosters Who Want to Enhance Their Cloud Offerings with Cloud Storage

Moving Virtual Storage to the Cloud. Guidelines for Hosters Who Want to Enhance Their Cloud Offerings with Cloud Storage Moving Virtual Storage to the Cloud Guidelines for Hosters Who Want to Enhance Their Cloud Offerings with Cloud Storage Table of Contents Overview... 1 Understanding the Storage Problem... 1 What Makes

More information

Moving Virtual Storage to the Cloud

Moving Virtual Storage to the Cloud Moving Virtual Storage to the Cloud White Paper Guidelines for Hosters Who Want to Enhance Their Cloud Offerings with Cloud Storage www.parallels.com Table of Contents Overview... 3 Understanding the Storage

More information

Journal of science STUDY ON REPLICA MANAGEMENT AND HIGH AVAILABILITY IN HADOOP DISTRIBUTED FILE SYSTEM (HDFS)

Journal of science STUDY ON REPLICA MANAGEMENT AND HIGH AVAILABILITY IN HADOOP DISTRIBUTED FILE SYSTEM (HDFS) Journal of science e ISSN 2277-3290 Print ISSN 2277-3282 Information Technology www.journalofscience.net STUDY ON REPLICA MANAGEMENT AND HIGH AVAILABILITY IN HADOOP DISTRIBUTED FILE SYSTEM (HDFS) S. Chandra

More information

Jeffrey D. Ullman slides. MapReduce for data intensive computing

Jeffrey D. Ullman slides. MapReduce for data intensive computing Jeffrey D. Ullman slides MapReduce for data intensive computing Single-node architecture CPU Machine Learning, Statistics Memory Classical Data Mining Disk Commodity Clusters Web data sets can be very

More information

Quantum StorNext. Product Brief: Distributed LAN Client

Quantum StorNext. Product Brief: Distributed LAN Client Quantum StorNext Product Brief: Distributed LAN Client NOTICE This product brief may contain proprietary information protected by copyright. Information in this product brief is subject to change without

More information

1. Comments on reviews a. Need to avoid just summarizing web page asks you for:

1. Comments on reviews a. Need to avoid just summarizing web page asks you for: 1. Comments on reviews a. Need to avoid just summarizing web page asks you for: i. A one or two sentence summary of the paper ii. A description of the problem they were trying to solve iii. A summary of

More information

Backup Solutions with Open-E Data Storage Software (DSS V6)

Backup Solutions with Open-E Data Storage Software (DSS V6) Solutions with Open-E Data Storage Software (DSS V6) Software Version: DSS ver. 6.00 up40 Presentation updated: September 2010 Different s opportunities using Open-E DSS The storage market is still growing

More information

Lecture 3: Scaling by Load Balancing 1. Comments on reviews i. 2. Topic 1: Scalability a. QUESTION: What are problems? i. These papers look at

Lecture 3: Scaling by Load Balancing 1. Comments on reviews i. 2. Topic 1: Scalability a. QUESTION: What are problems? i. These papers look at Lecture 3: Scaling by Load Balancing 1. Comments on reviews i. 2. Topic 1: Scalability a. QUESTION: What are problems? i. These papers look at distributing load b. QUESTION: What is the context? i. How

More information

V:Drive - Costs and Benefits of an Out-of-Band Storage Virtualization System

V:Drive - Costs and Benefits of an Out-of-Band Storage Virtualization System V:Drive - Costs and Benefits of an Out-of-Band Storage Virtualization System André Brinkmann, Michael Heidebuer, Friedhelm Meyer auf der Heide, Ulrich Rückert, Kay Salzwedel, and Mario Vodisek Paderborn

More information

Web Email DNS Peer-to-peer systems (file sharing, CDNs, cycle sharing)

Web Email DNS Peer-to-peer systems (file sharing, CDNs, cycle sharing) 1 1 Distributed Systems What are distributed systems? How would you characterize them? Components of the system are located at networked computers Cooperate to provide some service No shared memory Communication

More information

Scala Storage Scale-Out Clustered Storage White Paper

Scala Storage Scale-Out Clustered Storage White Paper White Paper Scala Storage Scale-Out Clustered Storage White Paper Chapter 1 Introduction... 3 Capacity - Explosive Growth of Unstructured Data... 3 Performance - Cluster Computing... 3 Chapter 2 Current

More information

Distributed File Systems

Distributed File Systems Distributed File Systems Alemnew Sheferaw Asrese University of Trento - Italy December 12, 2012 Acknowledgement: Mauro Fruet Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 1 / 55 Outline

More information

Agenda. Enterprise Application Performance Factors. Current form of Enterprise Applications. Factors to Application Performance.

Agenda. Enterprise Application Performance Factors. Current form of Enterprise Applications. Factors to Application Performance. Agenda Enterprise Performance Factors Overall Enterprise Performance Factors Best Practice for generic Enterprise Best Practice for 3-tiers Enterprise Hardware Load Balancer Basic Unix Tuning Performance

More information

PARALLELS CLOUD STORAGE

PARALLELS CLOUD STORAGE PARALLELS CLOUD STORAGE Performance Benchmark Results 1 Table of Contents Executive Summary... Error! Bookmark not defined. Architecture Overview... 3 Key Features... 5 No Special Hardware Requirements...

More information

Index Terms : Load rebalance, distributed file systems, clouds, movement cost, load imbalance, chunk.

Index Terms : Load rebalance, distributed file systems, clouds, movement cost, load imbalance, chunk. Load Rebalancing for Distributed File Systems in Clouds. Smita Salunkhe, S. S. Sannakki Department of Computer Science and Engineering KLS Gogte Institute of Technology, Belgaum, Karnataka, India Affiliated

More information

Big Data Processing in the Cloud. Shadi Ibrahim Inria, Rennes - Bretagne Atlantique Research Center

Big Data Processing in the Cloud. Shadi Ibrahim Inria, Rennes - Bretagne Atlantique Research Center Big Data Processing in the Cloud Shadi Ibrahim Inria, Rennes - Bretagne Atlantique Research Center Data is ONLY as useful as the decisions it enables 2 Data is ONLY as useful as the decisions it enables

More information

Storage Networking Overview

Storage Networking Overview Networking Overview iscsi Attached LAN Networking SAN NAS Gateway NAS Attached SAN Attached IBM Total Module Flow Business Challenges Networking Trends and Directions What is Networking? Technological

More information

Distributed File Systems

Distributed File Systems Distributed File Systems File Characteristics From Andrew File System work: most files are small transfer files rather than disk blocks? reading more common than writing most access is sequential most

More information

Overview of I/O Performance and RAID in an RDBMS Environment. By: Edward Whalen Performance Tuning Corporation

Overview of I/O Performance and RAID in an RDBMS Environment. By: Edward Whalen Performance Tuning Corporation Overview of I/O Performance and RAID in an RDBMS Environment By: Edward Whalen Performance Tuning Corporation Abstract This paper covers the fundamentals of I/O topics and an overview of RAID levels commonly

More information

The Hadoop Distributed File System

The Hadoop Distributed File System The Hadoop Distributed File System The Hadoop Distributed File System, Konstantin Shvachko, Hairong Kuang, Sanjay Radia, Robert Chansler, Yahoo, 2010 Agenda Topic 1: Introduction Topic 2: Architecture

More information

Cloud Storage. Parallels. Performance Benchmark Results. White Paper. www.parallels.com

Cloud Storage. Parallels. Performance Benchmark Results. White Paper. www.parallels.com Parallels Cloud Storage White Paper Performance Benchmark Results www.parallels.com Table of Contents Executive Summary... 3 Architecture Overview... 3 Key Features... 4 No Special Hardware Requirements...

More information

Data Center Design & Virtualization. Md. Jahangir Hossain Open Communication Limited jahangir@open.com.bd

Data Center Design & Virtualization. Md. Jahangir Hossain Open Communication Limited jahangir@open.com.bd Data Center Design & Virtualization Md. Jahangir Hossain Open Communication Limited jahangir@open.com.bd Objectives Data Center Architecture Data Center Standard Data Center Design Model Application Design

More information

Big Table A Distributed Storage System For Data

Big Table A Distributed Storage System For Data Big Table A Distributed Storage System For Data OSDI 2006 Fay Chang, Jeffrey Dean, Sanjay Ghemawat et.al. Presented by Rahul Malviya Why BigTable? Lots of (semi-)structured data at Google - - URLs: Contents,

More information

POWER ALL GLOBAL FILE SYSTEM (PGFS)

POWER ALL GLOBAL FILE SYSTEM (PGFS) POWER ALL GLOBAL FILE SYSTEM (PGFS) Defining next generation of global storage grid Power All Networks Ltd. Technical Whitepaper April 2008, version 1.01 Table of Content 1. Introduction.. 3 2. Paradigm

More information

SAN Conceptual and Design Basics

SAN Conceptual and Design Basics TECHNICAL NOTE VMware Infrastructure 3 SAN Conceptual and Design Basics VMware ESX Server can be used in conjunction with a SAN (storage area network), a specialized high speed network that connects computer

More information

CHAPTER 17: File Management

CHAPTER 17: File Management CHAPTER 17: File Management The Architecture of Computer Hardware, Systems Software & Networking: An Information Technology Approach 4th Edition, Irv Englander John Wiley and Sons 2010 PowerPoint slides

More information

Sunita Suralkar, Ashwini Mujumdar, Gayatri Masiwal, Manasi Kulkarni Department of Computer Technology, Veermata Jijabai Technological Institute

Sunita Suralkar, Ashwini Mujumdar, Gayatri Masiwal, Manasi Kulkarni Department of Computer Technology, Veermata Jijabai Technological Institute Review of Distributed File Systems: Case Studies Sunita Suralkar, Ashwini Mujumdar, Gayatri Masiwal, Manasi Kulkarni Department of Computer Technology, Veermata Jijabai Technological Institute Abstract

More information

STORAGE. 2015 Arka Service s.r.l.

STORAGE. 2015 Arka Service s.r.l. STORAGE STORAGE MEDIA independently from the repository model used, data must be saved on a support (data storage media). Arka Service uses the most common methods used as market standard such as: MAGNETIC

More information

Distributed Data Stores

Distributed Data Stores Distributed Data Stores 1 Distributed Persistent State MapReduce addresses distributed processing of aggregation-based queries Persistent state across a large number of machines? Distributed DBMS High

More information

OPTIMIZING EXCHANGE SERVER IN A TIERED STORAGE ENVIRONMENT WHITE PAPER NOVEMBER 2006

OPTIMIZING EXCHANGE SERVER IN A TIERED STORAGE ENVIRONMENT WHITE PAPER NOVEMBER 2006 OPTIMIZING EXCHANGE SERVER IN A TIERED STORAGE ENVIRONMENT WHITE PAPER NOVEMBER 2006 EXECUTIVE SUMMARY Microsoft Exchange Server is a disk-intensive application that requires high speed storage to deliver

More information

THE HADOOP DISTRIBUTED FILE SYSTEM

THE HADOOP DISTRIBUTED FILE SYSTEM THE HADOOP DISTRIBUTED FILE SYSTEM Konstantin Shvachko, Hairong Kuang, Sanjay Radia, Robert Chansler Presented by Alexander Pokluda October 7, 2013 Outline Motivation and Overview of Hadoop Architecture,

More information

Ultimate Guide to Oracle Storage

Ultimate Guide to Oracle Storage Ultimate Guide to Oracle Storage Presented by George Trujillo George.Trujillo@trubix.com George Trujillo Twenty two years IT experience with 19 years Oracle experience. Advanced database solutions such

More information

Hypertable Architecture Overview

Hypertable Architecture Overview WHITE PAPER - MARCH 2012 Hypertable Architecture Overview Hypertable is an open source, scalable NoSQL database modeled after Bigtable, Google s proprietary scalable database. It is written in C++ for

More information

Seminar Presentation for ECE 658 Instructed by: Prof.Anura Jayasumana Distributed File Systems

Seminar Presentation for ECE 658 Instructed by: Prof.Anura Jayasumana Distributed File Systems Seminar Presentation for ECE 658 Instructed by: Prof.Anura Jayasumana Distributed File Systems Prabhakaran Murugesan Outline File Transfer Protocol (FTP) Network File System (NFS) Andrew File System (AFS)

More information

Survey on Load Rebalancing for Distributed File System in Cloud

Survey on Load Rebalancing for Distributed File System in Cloud Survey on Load Rebalancing for Distributed File System in Cloud Prof. Pranalini S. Ketkar Ankita Bhimrao Patkure IT Department, DCOER, PG Scholar, Computer Department DCOER, Pune University Pune university

More information

Technology Insight Series

Technology Insight Series Evaluating Storage Technologies for Virtual Server Environments Russ Fellows June, 2010 Technology Insight Series Evaluator Group Copyright 2010 Evaluator Group, Inc. All rights reserved Executive Summary

More information

Algorithms and Methods for Distributed Storage Networks 9 Analysis of DHT Christian Schindelhauer

Algorithms and Methods for Distributed Storage Networks 9 Analysis of DHT Christian Schindelhauer Algorithms and Methods for 9 Analysis of DHT Institut für Informatik Wintersemester 2007/08 Distributed Hash-Table (DHT) Hash table does not work efficiently for inserting and deleting Distributed Hash-Table

More information

Archive Data Retention & Compliance. Solutions Integrated Storage Appliances. Management Optimized Storage & Migration

Archive Data Retention & Compliance. Solutions Integrated Storage Appliances. Management Optimized Storage & Migration Solutions Integrated Storage Appliances Management Optimized Storage & Migration Archive Data Retention & Compliance Services Global Installation & Support SECURING THE FUTURE OF YOUR DATA w w w.q sta

More information

Using EonStor FC-host Storage Systems in VMware Infrastructure 3 and vsphere 4

Using EonStor FC-host Storage Systems in VMware Infrastructure 3 and vsphere 4 Using EonStor FC-host Storage Systems in VMware Infrastructure 3 and vsphere 4 Application Note Abstract This application note explains the configure details of using Infortrend FC-host storage systems

More information

Storage Backup and Disaster Recovery: Using New Technology to Develop Best Practices

Storage Backup and Disaster Recovery: Using New Technology to Develop Best Practices Storage Backup and Disaster Recovery: Using New Technology to Develop Best Practices September 2008 Recent advances in data storage and data protection technology are nothing short of phenomenal. Today,

More information

Distributed File System Choices: Red Hat Storage, GFS2 & pnfs

Distributed File System Choices: Red Hat Storage, GFS2 & pnfs Distributed File System Choices: Red Hat Storage, GFS2 & pnfs Ric Wheeler Architect & Senior Manager, Red Hat June 27, 2012 Overview Distributed file system basics Red Hat distributed file systems Performance

More information

www.basho.com Technical Overview Simple, Scalable, Object Storage Software

www.basho.com Technical Overview Simple, Scalable, Object Storage Software www.basho.com Technical Overview Simple, Scalable, Object Storage Software Table of Contents Table of Contents... 1 Introduction & Overview... 1 Architecture... 2 How it Works... 2 APIs and Interfaces...

More information

The Sierra Clustered Database Engine, the technology at the heart of

The Sierra Clustered Database Engine, the technology at the heart of A New Approach: Clustrix Sierra Database Engine The Sierra Clustered Database Engine, the technology at the heart of the Clustrix solution, is a shared-nothing environment that includes the Sierra Parallel

More information

Large Scale Storage. Orlando Richards, Information Services orlando.richards@ed.ac.uk. LCFG Users Day, University of Edinburgh 18 th January 2013

Large Scale Storage. Orlando Richards, Information Services orlando.richards@ed.ac.uk. LCFG Users Day, University of Edinburgh 18 th January 2013 Large Scale Storage Orlando Richards, Information Services orlando.richards@ed.ac.uk LCFG Users Day, University of Edinburgh 18 th January 2013 Overview My history of storage services What is (and is not)

More information

VMware vsphere Data Protection 6.0

VMware vsphere Data Protection 6.0 VMware vsphere Data Protection 6.0 TECHNICAL OVERVIEW REVISED FEBRUARY 2015 Table of Contents Introduction.... 3 Architectural Overview... 4 Deployment and Configuration.... 5 Backup.... 6 Application

More information

DELL RAID PRIMER DELL PERC RAID CONTROLLERS. Joe H. Trickey III. Dell Storage RAID Product Marketing. John Seward. Dell Storage RAID Engineering

DELL RAID PRIMER DELL PERC RAID CONTROLLERS. Joe H. Trickey III. Dell Storage RAID Product Marketing. John Seward. Dell Storage RAID Engineering DELL RAID PRIMER DELL PERC RAID CONTROLLERS Joe H. Trickey III Dell Storage RAID Product Marketing John Seward Dell Storage RAID Engineering http://www.dell.com/content/topics/topic.aspx/global/products/pvaul/top

More information

Ceph. A file system a little bit different. Udo Seidel

Ceph. A file system a little bit different. Udo Seidel Ceph A file system a little bit different Udo Seidel Ceph what? So-called parallel distributed cluster file system Started as part of PhD studies at UCSC Public announcement in 2006 at 7 th OSDI File system

More information

MinCopysets: Derandomizing Replication In Cloud Storage

MinCopysets: Derandomizing Replication In Cloud Storage MinCopysets: Derandomizing Replication In Cloud Storage Asaf Cidon, Ryan Stutsman, Stephen Rumble, Sachin Katti, John Ousterhout and Mendel Rosenblum Stanford University cidon@stanford.edu, {stutsman,rumble,skatti,ouster,mendel}@cs.stanford.edu

More information

CASE STUDY: Oracle TimesTen In-Memory Database and Shared Disk HA Implementation at Instance level. -ORACLE TIMESTEN 11gR1

CASE STUDY: Oracle TimesTen In-Memory Database and Shared Disk HA Implementation at Instance level. -ORACLE TIMESTEN 11gR1 CASE STUDY: Oracle TimesTen In-Memory Database and Shared Disk HA Implementation at Instance level -ORACLE TIMESTEN 11gR1 CASE STUDY Oracle TimesTen In-Memory Database and Shared Disk HA Implementation

More information

Oracle Database Deployments with EMC CLARiiON AX4 Storage Systems

Oracle Database Deployments with EMC CLARiiON AX4 Storage Systems Oracle Database Deployments with EMC CLARiiON AX4 Storage Systems Applied Technology Abstract This white paper investigates configuration and replication choices for Oracle Database deployment with EMC

More information

IOmark- VDI. Nimbus Data Gemini Test Report: VDI- 130906- a Test Report Date: 6, September 2013. www.iomark.org

IOmark- VDI. Nimbus Data Gemini Test Report: VDI- 130906- a Test Report Date: 6, September 2013. www.iomark.org IOmark- VDI Nimbus Data Gemini Test Report: VDI- 130906- a Test Copyright 2010-2013 Evaluator Group, Inc. All rights reserved. IOmark- VDI, IOmark- VDI, VDI- IOmark, and IOmark are trademarks of Evaluator

More information

A Content-Based Load Balancing Algorithm for Metadata Servers in Cluster File Systems*

A Content-Based Load Balancing Algorithm for Metadata Servers in Cluster File Systems* A Content-Based Load Balancing Algorithm for Metadata Servers in Cluster File Systems* Junho Jang, Saeyoung Han, Sungyong Park, and Jihoon Yang Department of Computer Science and Interdisciplinary Program

More information

Table of contents. Matching server virtualization with advanced storage virtualization

Table of contents. Matching server virtualization with advanced storage virtualization Matching server virtualization with advanced storage virtualization Using HP LeftHand SAN and VMware Infrastructure 3 for improved ease of use, reduced cost and complexity, increased availability, and

More information

Hadoop Distributed File System. T-111.5550 Seminar On Multimedia 2009-11-11 Eero Kurkela

Hadoop Distributed File System. T-111.5550 Seminar On Multimedia 2009-11-11 Eero Kurkela Hadoop Distributed File System T-111.5550 Seminar On Multimedia 2009-11-11 Eero Kurkela Agenda Introduction Flesh and bones of HDFS Architecture Accessing data Data replication strategy Fault tolerance

More information

Big data management with IBM General Parallel File System

Big data management with IBM General Parallel File System Big data management with IBM General Parallel File System Optimize storage management and boost your return on investment Highlights Handles the explosive growth of structured and unstructured data Offers

More information

Storage Architectures for Big Data in the Cloud

Storage Architectures for Big Data in the Cloud Storage Architectures for Big Data in the Cloud Sam Fineberg HP Storage CT Office/ May 2013 Overview Introduction What is big data? Big Data I/O Hadoop/HDFS SAN Distributed FS Cloud Summary Research Areas

More information

WHITE PAPER. Permabit Albireo Data Optimization Software. Benefits of Albireo for Virtual Servers. January 2012. Permabit Technology Corporation

WHITE PAPER. Permabit Albireo Data Optimization Software. Benefits of Albireo for Virtual Servers. January 2012. Permabit Technology Corporation WHITE PAPER Permabit Albireo Data Optimization Software Benefits of Albireo for Virtual Servers January 2012 Permabit Technology Corporation Ten Canal Park Cambridge, MA 02141 USA Phone: 617.252.9600 FAX:

More information

Direct NFS - Design considerations for next-gen NAS appliances optimized for database workloads Akshay Shah Gurmeet Goindi Oracle

Direct NFS - Design considerations for next-gen NAS appliances optimized for database workloads Akshay Shah Gurmeet Goindi Oracle Direct NFS - Design considerations for next-gen NAS appliances optimized for database workloads Akshay Shah Gurmeet Goindi Oracle Agenda Introduction Database Architecture Direct NFS Client NFS Server

More information

Scale and Availability Considerations for Cluster File Systems. David Noy, Symantec Corporation

Scale and Availability Considerations for Cluster File Systems. David Noy, Symantec Corporation Scale and Availability Considerations for Cluster File Systems David Noy, Symantec Corporation SNIA Legal Notice The material contained in this tutorial is copyrighted by the SNIA unless otherwise noted.

More information

High Availability Storage

High Availability Storage High Availability Storage High Availability Extensions Goldwyn Rodrigues High Availability Storage Engineer SUSE High Availability Extensions Highly available services for mission critical systems Integrated

More information

Virtualization, Business Continuation Plan & Disaster Recovery for EMS -By Ramanj Pamidi San Diego Gas & Electric

Virtualization, Business Continuation Plan & Disaster Recovery for EMS -By Ramanj Pamidi San Diego Gas & Electric Virtualization, Business Continuation Plan & Disaster Recovery for EMS -By Ramanj Pamidi San Diego Gas & Electric 2001 San Diego Gas and Electric. All copyright and trademark rights reserved. Importance

More information

Block based, file-based, combination. Component based, solution based

Block based, file-based, combination. Component based, solution based The Wide Spread Role of 10-Gigabit Ethernet in Storage This paper provides an overview of SAN and NAS storage solutions, highlights the ubiquitous role of 10 Gigabit Ethernet in these solutions, and illustrates

More information

Windows Server 2012 R2 Hyper-V: Designing for the Real World

Windows Server 2012 R2 Hyper-V: Designing for the Real World Windows Server 2012 R2 Hyper-V: Designing for the Real World Steve Evans @scevans www.loudsteve.com Nick Hawkins @nhawkins www.nickahawkins.com Is Hyper-V for real? Microsoft Fan Boys Reality VMware Hyper-V

More information

VERITAS Storage Foundation 4.3 for Windows

VERITAS Storage Foundation 4.3 for Windows DATASHEET VERITAS Storage Foundation 4.3 for Windows Advanced Volume Management Technology for Windows In distributed client/server environments, users demand that databases, mission-critical applications

More information

DAS to SAN Migration Using a Storage Concentrator

DAS to SAN Migration Using a Storage Concentrator DAS to SAN Migration Using a Storage Concentrator April 2006 All trademark names are the property of their respective companies. This publication contains opinions of StoneFly, Inc. which are subject to

More information

VMware Virtual Machine File System: Technical Overview and Best Practices

VMware Virtual Machine File System: Technical Overview and Best Practices VMware Virtual Machine File System: Technical Overview and Best Practices A VMware Technical White Paper Version 1.0. VMware Virtual Machine File System: Technical Overview and Best Practices Paper Number:

More information

CS2510 Computer Operating Systems

CS2510 Computer Operating Systems CS2510 Computer Operating Systems HADOOP Distributed File System Dr. Taieb Znati Computer Science Department University of Pittsburgh Outline HDF Design Issues HDFS Application Profile Block Abstraction

More information

CS2510 Computer Operating Systems

CS2510 Computer Operating Systems CS2510 Computer Operating Systems HADOOP Distributed File System Dr. Taieb Znati Computer Science Department University of Pittsburgh Outline HDF Design Issues HDFS Application Profile Block Abstraction

More information

A Brief Analysis on Architecture and Reliability of Cloud Based Data Storage

A Brief Analysis on Architecture and Reliability of Cloud Based Data Storage Volume 2, No.4, July August 2013 International Journal of Information Systems and Computer Sciences ISSN 2319 7595 Tejaswini S L Jayanthy et al., Available International Online Journal at http://warse.org/pdfs/ijiscs03242013.pdf

More information

Running a Workflow on a PowerCenter Grid

Running a Workflow on a PowerCenter Grid Running a Workflow on a PowerCenter Grid 2010-2014 Informatica Corporation. No part of this document may be reproduced or transmitted in any form, by any means (electronic, photocopying, recording or otherwise)

More information

Storage and High Availability with Windows Server 10971B; 4 Days, Instructor-led

Storage and High Availability with Windows Server 10971B; 4 Days, Instructor-led Storage and High Availability with Windows Server 10971B; 4 Days, Instructor-led Course Description Get hands-on instruction and practice provisioning your storage requirements and meeting your high availability

More information

Cluster Computing. ! Fault tolerance. ! Stateless. ! Throughput. ! Stateful. ! Response time. Architectures. Stateless vs. Stateful.

Cluster Computing. ! Fault tolerance. ! Stateless. ! Throughput. ! Stateful. ! Response time. Architectures. Stateless vs. Stateful. Architectures Cluster Computing Job Parallelism Request Parallelism 2 2010 VMware Inc. All rights reserved Replication Stateless vs. Stateful! Fault tolerance High availability despite failures If one

More information

Backup architectures in the modern data center. Author: Edmond van As edmond@competa.com Competa IT b.v.

Backup architectures in the modern data center. Author: Edmond van As edmond@competa.com Competa IT b.v. Backup architectures in the modern data center. Author: Edmond van As edmond@competa.com Competa IT b.v. Existing backup methods Most companies see an explosive growth in the amount of data that they have

More information

Distributed Metadata Management Scheme in HDFS

Distributed Metadata Management Scheme in HDFS International Journal of Scientific and Research Publications, Volume 3, Issue 5, May 2013 1 Distributed Metadata Management Scheme in HDFS Mrudula Varade *, Vimla Jethani ** * Department of Computer Engineering,

More information

File System & Device Drive. Overview of Mass Storage Structure. Moving head Disk Mechanism. HDD Pictures 11/13/2014. CS341: Operating System

File System & Device Drive. Overview of Mass Storage Structure. Moving head Disk Mechanism. HDD Pictures 11/13/2014. CS341: Operating System CS341: Operating System Lect 36: 1 st Nov 2014 Dr. A. Sahu Dept of Comp. Sc. & Engg. Indian Institute of Technology Guwahati File System & Device Drive Mass Storage Disk Structure Disk Arm Scheduling RAID

More information

Dell Compellent Storage Center SAN & VMware View 1,000 Desktop Reference Architecture. Dell Compellent Product Specialist Team

Dell Compellent Storage Center SAN & VMware View 1,000 Desktop Reference Architecture. Dell Compellent Product Specialist Team Dell Compellent Storage Center SAN & VMware View 1,000 Desktop Reference Architecture Dell Compellent Product Specialist Team THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL

More information

Hadoop & its Usage at Facebook

Hadoop & its Usage at Facebook Hadoop & its Usage at Facebook Dhruba Borthakur Project Lead, Hadoop Distributed File System dhruba@apache.org Presented at the Storage Developer Conference, Santa Clara September 15, 2009 Outline Introduction

More information

Storage Networking Foundations Certification Workshop

Storage Networking Foundations Certification Workshop Storage Networking Foundations Certification Workshop Duration: 2 Days Type: Lecture Course Description / Overview / Expected Outcome A group of students was asked recently to define a "SAN." Some replies

More information

Dependable Systems. 9. Redundant arrays of. Prof. Dr. Miroslaw Malek. Wintersemester 2004/05 www.informatik.hu-berlin.de/rok/zs

Dependable Systems. 9. Redundant arrays of. Prof. Dr. Miroslaw Malek. Wintersemester 2004/05 www.informatik.hu-berlin.de/rok/zs Dependable Systems 9. Redundant arrays of inexpensive disks (RAID) Prof. Dr. Miroslaw Malek Wintersemester 2004/05 www.informatik.hu-berlin.de/rok/zs Redundant Arrays of Inexpensive Disks (RAID) RAID is

More information

DATA COMMUNICATOIN NETWORKING

DATA COMMUNICATOIN NETWORKING DATA COMMUNICATOIN NETWORKING Instructor: Ouldooz Baghban Karimi Course Book: Computer Networking, A Top-Down Approach, Kurose, Ross Slides: - Course book Slides - Slides from Princeton University COS461

More information

VMware Virtual SAN Backup Using VMware vsphere Data Protection Advanced SEPTEMBER 2014

VMware Virtual SAN Backup Using VMware vsphere Data Protection Advanced SEPTEMBER 2014 VMware SAN Backup Using VMware vsphere Data Protection Advanced SEPTEMBER 2014 VMware SAN Backup Using VMware vsphere Table of Contents Introduction.... 3 vsphere Architectural Overview... 4 SAN Backup

More information

Load Rebalancing for File System in Public Cloud Roopa R.L 1, Jyothi Patil 2

Load Rebalancing for File System in Public Cloud Roopa R.L 1, Jyothi Patil 2 Load Rebalancing for File System in Public Cloud Roopa R.L 1, Jyothi Patil 2 1 PDA College of Engineering, Gulbarga, Karnataka, India rlrooparl@gmail.com 2 PDA College of Engineering, Gulbarga, Karnataka,

More information

THE CLOUD STORAGE ARGUMENT

THE CLOUD STORAGE ARGUMENT THE CLOUD STORAGE ARGUMENT The argument over the right type of storage for data center applications is an ongoing battle. This argument gets amplified when discussing cloud architectures both private and

More information

Optimizing Large Arrays with StoneFly Storage Concentrators

Optimizing Large Arrays with StoneFly Storage Concentrators Optimizing Large Arrays with StoneFly Storage Concentrators All trademark names are the property of their respective companies. This publication contains opinions of which are subject to change from time

More information