COSC 6374 Parallel Computation. Parallel I/O (I) I/O basics. Concept of a clusters

Size: px
Start display at page:

Download "COSC 6374 Parallel Computation. Parallel I/O (I) I/O basics. Concept of a clusters"

Transcription

1 COSC 6374 Parallel I/O (I) I/O basics Fall 2012 Concept of a clusters Processor 1 local disks Compute node message passing network administrative network Memory Processor 2 Network card 1 Network card 2 1

2 I/O Problem (I) Every node has its own local disk Most applications require data and executable to be locally available e.g. an MPI application using multiple nodes requires executable to be available on all nodes in the same directory using the same name Multiple processes need to access the same file potentially different portions efficiency Basic characteristics of storage devices Capacity: amount of data a device can store Transfer rate or bandwidth: amount of data at which a device can read/write in a certain amount of time Access time or latency: delay before the first byte is moved Prefix Abbreviation Base ten Base two kilo, kibi K, Ki 10^3 2^10=1024 Mega, mebi M, Mi 10^6 2^20 Giga, gibi G, Gi 10^9 2^30 Tera, tebi T, Ti 10^12 2^40 Peta, pebi P, Pi 10^15 2^50 2

3 UNIX File Access Model A File is a sequence of bytes When a program opens a file, the file system establishes a file pointer. The file pointer is an integer indicating the position in the file, where the next byte will be written/read. Disk drives read and write data in fixed-sized units (disk sectors) File systems allocate space in blocks, which is a fixed number of contiguous disk sectors. In UNIX based file systems, the blocks that hold data are listed in an inode. An inode contains the information needed to find all the blocks that belong to a file. If a file is too large and an inode can not hold the whole list of blocks, intermediate nodes (indirect blocks) are introduced. Write operations Write: the file systems copies bytes from the user buffer into system buffer. If buffer filled up, system sends data to disk System buffering + allows file systems to collect full blocks of data before sending to disk + File system can send several blocks at once to the disk (delayed write or write behind) - Data not really saved in the case of a system crash - For very large write operations, the additional copy from user to system buffer could/should be avoided 3

4 Read operations Read: File system determines, which blocks contain requested data Read blocks from disk into system buffer Copy data from system buffer into user memory System buffering: + file system always reads a full block (file caching) + If application reads data sequentially, prefetching (read ahead) can improve performance - Prefetching harmful to the performance, if application has a random access pattern. Dealing with disk latency: Caching and buffering Avoids repeated access to the same block Allows a file system to smooth out I/O behavior Helps to hide the latency of the hard drives Lowers the performance of I/O operations for irregular access Non-blocking I/O gives users control over prefetching and delayed writing Initiate read/write operations as soon as possible Wait for the finishing of the read/write operations just when absolutely necessary. 4

5 Improving Disk Bandwidth: disk striping Utilize multiple hard drives Split a file into constant chunks and distribute them across all disks Three relevant parameters: Stripe factor: number of disks Stripe depth: size of each block Which disk contains the first block of the file Block 1 Block 2 Block 3 Block n Disk 1 Disk 2 Disk 3 Disk 4 Disk striping Ideal assumption b(n, p) = p * b(n/p, 1) with N: number of bytes to be written b: bandwidth p: number of disks Realistically: b(n,p) < p * b(n/p,1) since N is often not large enough to fully utilize p hard drives networking overhead 5

6 Two levels of disk striping (I) Using a RAID controller Hardware typically a single box number of disks: 3 n Redundant arrays of independent disks (RAID) Goals: improve reliability and performance of an I/O system improve performance of an I/O system Several RAID levels defined RAID 0: disk striping without redundant storage ( JBOD = just a bunch of disks) No fault tolerance Good for high transfer rates i.e. read/write bandwidth of a single large file Good for high request rates i.e. access time to many (small) files RAID 1: mirroring All data is replicated on two or more disks Does not improve write performance and just moderately the read performance 6

7 RAID level 2 RAID 2: Hamming codes Each group of data bits has several check bits appended to it forming Hamming code words Each bit of a Hamming code word is stored on a separate disk Very high additional costs: e.g. up to 50% additional capacity required Hardly used today since parity based codes faster and easier RAID level 3 Parity based protection: Based on exclusive OR (XOR) Reversible Example (data byte 1) XOR (data byte 2) (parity byte) Recovery (data byte 2) XOR (parity byte) (recovered data byte 1) 7

8 RAID level 3 (cont.) Data divided evenly into N subblocks (N = number of disks, typically 4 or 5) Computing parity bytes generates an additional subblock Subblocks written in parallel on N+1 disks For best performance data should be of size (N * sector size) Problems with RAID level 3: All disks are always participating in every operation => contention for applications with high access rates If data size is less than N*sector size, system has to read old subblocks to calculate the parity bytes RAID level 3 good for high transfer rates RAID level 4 Parity bytes for N disks calculated and stored Parity bytes are stored on a separate disk Files are not necessarily distributed over N disks For read operations: Determine disks for the requested blocks Read data from these disks For write operations Retrieve the old data from the sector being overwritten Retrieve parity block from the parity disk Extract old data from the parity block using XOR operations Add the new data to the parity block using XOR Store new data Store new parity block Bottleneck: parity disk is involved in every operation 8

9 RAID level 5 Same as RAID 4, but parity blocks are distributed on different disks Block 1 Block 2 Block 3 Block 4 P(1,2,3,4) Block 5 Block 6 Block 7 P(5,6,7,8) Block 8 RAID level 6 Tolerates the loss of more than one disk Collection of several techniques E.g. P+Q parity: store parity bytes using two different algorithms and store the two parity blocks on different disks E.g. Two dimensional parity Parity disks 9

10 Is RAID level 1 + RAID level 0 RAID 1 mirroring RAID level 10 RAID 0 striping Also available: RAID 53 (RAID 0 + RAID 3) Comparing RAID levels RAID level Protection Space usage Good at.. Poor at.. 0 None N Performance Data protect. 1 Mirroring 2N Data protect. Space effic. 2 Hamming codes ~1.5N Transfer rate Request rate 3 Parity N+1 Transfer rate Request rate 4 Parity N+1 Read req. rate Write perf. 5 Parity N+1 Request rate Transfer rate 6 P+Q or 2-D (N+2) or (MN+M+N) Data protect. Write perf. 10 Mirroring 2N Performance Space effic. 53 parity N+striping factor Performance Space effic. 10

11 Two levels of disk striping (II) Using a parallel file system exposes the individual units capable of handling data often called storage servers, I/O nodes, etc. each storage server might use multiple hard drives underneath the hood to increase its read/write bandwidth Metadata server which keeps track of which parts of a file are on which storage server Single disk failure less of a problem, if each server uses underneath the hood a RAID 5 storage system Parallel File Systems: Conceptual overview Compute nodes Meta-data server storage server 0 storage server 1 storage server 2 storage server 3 11

12 File access on a parallel file system Compute node Metadata server Application calls write() OS requests list of relevant I/O nodes for this write operation MD server sends storage IDs, offsets etc. OS sends data to storage servers Disk striping Requirements to improve performance of I/O operations using disk striping: Multiple physical disks Have to balance network bandwidth and I/O bandwidth Problem of simple disk striping: for a fixed file size, the number of disks which can be used in parallel is limited Prominent parallel file systems PVFS2 Lustre GPFS NFS v4.2 (new standard currently being ratified) 12

13 Distributed vs. Parallel File Systems Distributed File Systems Offer access to a collection of files on remote machines Typically client-server based approach Transparent for the user NFS The Network File System Protocol for a remote file service Stateless server (v3) Communication based on RPC (Remote Procedure Call) NFS provides session semantics changes to an open file are initially only visible to the process that modified the file File locking not part of NFS protocol (v3) but often available through a separate protocol/daemon Client caching not part of the NFS protocol (v3) implementation dependent behavior Network File System (NFS) Compute node = NFS client NFS server Application calls write() OS forwards data to NSF server NFS daemon receives data NFS daemon calls write() 13

14 Parallel vs. Distributed File Systems In distributed file systems: Concurrent access to the same file from several processes is considered to be an unlikely event Assume different (i.e. lower) numbers of processes accessing a file Different security requirements 14

COSC 6374 Parallel Computation. Parallel I/O (I) I/O basics. Concept of a clusters

COSC 6374 Parallel Computation. Parallel I/O (I) I/O basics. Concept of a clusters COSC 6374 Parallel Computation Parallel I/O (I) I/O basics Spring 2008 Concept of a clusters Processor 1 local disks Compute node message passing network administrative network Memory Processor 2 Network

More information

Storing Data: Disks and Files

Storing Data: Disks and Files Storing Data: Disks and Files (From Chapter 9 of textbook) Storing and Retrieving Data Database Management Systems need to: Store large volumes of data Store data reliably (so that data is not lost!) Retrieve

More information

Chapter 11 Distributed File Systems. Distributed File Systems

Chapter 11 Distributed File Systems. Distributed File Systems Chapter 11 Distributed File Systems Introduction Case studies NFS Coda 1 Distributed File Systems A distributed file system enables clients to access files stored on one or more remote file servers A file

More information

Network File System (NFS) Pradipta De pradipta.de@sunykorea.ac.kr

Network File System (NFS) Pradipta De pradipta.de@sunykorea.ac.kr Network File System (NFS) Pradipta De pradipta.de@sunykorea.ac.kr Today s Topic Network File System Type of Distributed file system NFS protocol NFS cache consistency issue CSE506: Ext Filesystem 2 NFS

More information

Network Attached Storage. Jinfeng Yang Oct/19/2015

Network Attached Storage. Jinfeng Yang Oct/19/2015 Network Attached Storage Jinfeng Yang Oct/19/2015 Outline Part A 1. What is the Network Attached Storage (NAS)? 2. What are the applications of NAS? 3. The benefits of NAS. 4. NAS s performance (Reliability

More information

Lecture 36: Chapter 6

Lecture 36: Chapter 6 Lecture 36: Chapter 6 Today s topic RAID 1 RAID Redundant Array of Inexpensive (Independent) Disks Use multiple smaller disks (c.f. one large disk) Parallelism improves performance Plus extra disk(s) for

More information

Outline. Database Management and Tuning. Overview. Hardware Tuning. Johann Gamper. Unit 12

Outline. Database Management and Tuning. Overview. Hardware Tuning. Johann Gamper. Unit 12 Outline Database Management and Tuning Hardware Tuning Johann Gamper 1 Free University of Bozen-Bolzano Faculty of Computer Science IDSE Unit 12 2 3 Conclusion Acknowledgements: The slides are provided

More information

Mass-Storage Devices: Disks. CSCI 5103 Operating Systems. Basic Disk Functionality. Magnetic Disks

Mass-Storage Devices: Disks. CSCI 5103 Operating Systems. Basic Disk Functionality. Magnetic Disks Mass-Storage Devices: Disks CSCI 5103 Operating Systems Instructor: Abhishek Chandra Disk Structure and Attachment Disk Scheduling Disk Management RAID Structure Stable Storage 2 Magnetic Disks Most common

More information

Chapter 6. 6.1 Introduction. Storage and Other I/O Topics. p. 570( 頁 585) Fig. 6.1. I/O devices can be characterized by. I/O bus connections

Chapter 6. 6.1 Introduction. Storage and Other I/O Topics. p. 570( 頁 585) Fig. 6.1. I/O devices can be characterized by. I/O bus connections Chapter 6 Storage and Other I/O Topics 6.1 Introduction I/O devices can be characterized by Behavior: input, output, storage Partner: human or machine Data rate: bytes/sec, transfers/sec I/O bus connections

More information

Review. Lecture 21: Reliable, High Performance Storage. Overview. Basic Disk & File System properties CSC 468 / CSC 2204 11/23/2006

Review. Lecture 21: Reliable, High Performance Storage. Overview. Basic Disk & File System properties CSC 468 / CSC 2204 11/23/2006 S 468 / S 2204 Review Lecture 2: Reliable, High Performance Storage S 469HF Fall 2006 ngela emke rown We ve looked at fault tolerance via server replication ontinue operating with up to f failures Recovery

More information

CS 153 Design of Operating Systems Spring 2015

CS 153 Design of Operating Systems Spring 2015 CS 153 Design of Operating Systems Spring 2015 Lecture 22: File system optimizations Physical Disk Structure Disk components Platters Surfaces Tracks Arm Track Sector Surface Sectors Cylinders Arm Heads

More information

Parallel IO. Single namespace. Performance. Disk locality awareness? Data integrity. Fault tolerance. Standard interface. Network of disks?

Parallel IO. Single namespace. Performance. Disk locality awareness? Data integrity. Fault tolerance. Standard interface. Network of disks? PARALLEL IO Parallel IO Single namespace Network of disks? Performance Data replication Multiple I/O paths Disk locality awareness? Data integrity Multiple writers Locking? Fault tolerance Hardware failure

More information

High Performance Computing. Course Notes 2007-2008. High Performance Storage

High Performance Computing. Course Notes 2007-2008. High Performance Storage High Performance Computing Course Notes 2007-2008 2008 High Performance Storage Storage devices Primary storage: register (1 CPU cycle, a few ns) Cache (10-200 cycles, 0.02-0.5us) Main memory Local main

More information

Outline. Database Tuning. Disk Allocation Raw vs. Cooked Files. Overview. Hardware Tuning. Nikolaus Augsten. Unit 6 WS 2015/16

Outline. Database Tuning. Disk Allocation Raw vs. Cooked Files. Overview. Hardware Tuning. Nikolaus Augsten. Unit 6 WS 2015/16 Outline Database Tuning Hardware Tuning Nikolaus Augsten University of Salzburg Department of Computer Science Database Group Unit 6 WS 2015/16 1 2 3 Conclusion Adapted from Database Tuning by Dennis Shasha

More information

Filing Systems. Filing Systems

Filing Systems. Filing Systems Filing Systems At the outset we identified long-term storage as desirable characteristic of an OS. EG: On-line storage for an MIS. Convenience of not having to re-write programs. Sharing of data in an

More information

RAID. Tiffany Yu-Han Chen. # The performance of different RAID levels # read/write/reliability (fault-tolerant)/overhead

RAID. Tiffany Yu-Han Chen. # The performance of different RAID levels # read/write/reliability (fault-tolerant)/overhead RAID # The performance of different RAID levels # read/write/reliability (fault-tolerant)/overhead Tiffany Yu-Han Chen (These slides modified from Hao-Hua Chu National Taiwan University) RAID 0 - Striping

More information

Data Striping. data blocks. disk controller one I/O stream. disk controller. Gustavo Alonso. IKS. ETH Zürich Low level caching 1

Data Striping. data blocks. disk controller one I/O stream. disk controller. Gustavo Alonso. IKS. ETH Zürich Low level caching 1 Data Striping The idea behind data striping is to distribute data among several disks so that it can be accessed in parallel Data striping takes place at a low system level (it is not user driven) and

More information

200 Chapter 7. (This observation is reinforced and elaborated in Exercises 7.5 and 7.6, and the reader is urged to work through them.

200 Chapter 7. (This observation is reinforced and elaborated in Exercises 7.5 and 7.6, and the reader is urged to work through them. 200 Chapter 7 (This observation is reinforced and elaborated in Exercises 7.5 and 7.6, and the reader is urged to work through them.) 7.2 RAID Disks are potential bottlenecks for system performance and

More information

Chapter 11 I/O Management and Disk Scheduling

Chapter 11 I/O Management and Disk Scheduling Operating Systems: Internals and Design Principles, 6/E William Stallings Chapter 11 I/O Management and Disk Scheduling Dave Bremer Otago Polytechnic, NZ 2008, Prentice Hall I/O Devices Roadmap Organization

More information

Price/performance Modern Memory Hierarchy

Price/performance Modern Memory Hierarchy Lecture 21: Storage Administration Take QUIZ 15 over P&H 6.1-4, 6.8-9 before 11:59pm today Project: Cache Simulator, Due April 29, 2010 NEW OFFICE HOUR TIME: Tuesday 1-2, McKinley Last Time Exam discussion

More information

An Introduction to RAID. Giovanni Stracquadanio stracquadanio@dmi.unict.it www.dmi.unict.it/~stracquadanio

An Introduction to RAID. Giovanni Stracquadanio stracquadanio@dmi.unict.it www.dmi.unict.it/~stracquadanio An Introduction to RAID Giovanni Stracquadanio stracquadanio@dmi.unict.it www.dmi.unict.it/~stracquadanio Outline A definition of RAID An ensemble of RAIDs JBOD RAID 0...5 Configuring and testing a Linux

More information

Storing Data: Disks and Files. Disks and Files. Why Not Store Everything in Main Memory? Chapter 7

Storing Data: Disks and Files. Disks and Files. Why Not Store Everything in Main Memory? Chapter 7 Storing : Disks and Files Chapter 7 Yea, from the table of my memory I ll wipe away all trivial fond records. -- Shakespeare, Hamlet base Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Disks and

More information

CSE 120 Principles of Operating Systems

CSE 120 Principles of Operating Systems CSE 120 Principles of Operating Systems Fall 2004 Lecture 13: FFS, LFS, RAID Geoffrey M. Voelker Overview We ve looked at disks and file systems generically Now we re going to look at some example file

More information

CS 464/564 Introduction to Database Management System Instructor: Abdullah Mueen

CS 464/564 Introduction to Database Management System Instructor: Abdullah Mueen CS 464/564 Introduction to Database Management System Instructor: Abdullah Mueen LECTURE 14: DATA STORAGE AND REPRESENTATION Data Storage Memory Hierarchy Disks Fields, Records, Blocks Variable-length

More information

416 Distributed Systems

416 Distributed Systems 416 Distributed Systems RAID, Feb 5 2016 Thanks to Greg Ganger and Remzi Arapaci-Dusseau for slides Replacement Rates HPC1 COM1 COM2 Component % Component % Component % Hard drive 30.6 Power supply 34.8

More information

Operating Systems. Redundant Array of Inexpensive Disks (RAID) Thomas Ropars.

Operating Systems. Redundant Array of Inexpensive Disks (RAID) Thomas Ropars. 1 Operating Systems Redundant Array of Inexpensive Disks (RAID) Thomas Ropars thomas.ropars@imag.fr 2016 2 References The content of these lectures is inspired by: Operating Systems: Three Easy Pieces

More information

Database Management Systems

Database Management Systems 4411 Database Management Systems Acknowledgements and copyrights: these slides are a result of combination of notes and slides with contributions from: Michael Kiffer, Arthur Bernstein, Philip Lewis, Anestis

More information

Distributed File Systems

Distributed File Systems Distributed File Systems Paul Krzyzanowski Rutgers University October 28, 2012 1 Introduction The classic network file systems we examined, NFS, CIFS, AFS, Coda, were designed as client-server applications.

More information

Sistemas Operativos: Input/Output Disks

Sistemas Operativos: Input/Output Disks Sistemas Operativos: Input/Output Disks Pedro F. Souto (pfs@fe.up.pt) April 28, 2012 Topics Magnetic Disks RAID Solid State Disks Topics Magnetic Disks RAID Solid State Disks Magnetic Disk Construction

More information

File System Implementation II

File System Implementation II Introduction to Operating Systems File System Implementation II Performance, Recovery, Network File System John Franco Electrical Engineering and Computing Systems University of Cincinnati Review Block

More information

Input / Ouput devices. I/O Chapter 8. Goals & Constraints. Measures of Performance. Anatomy of a Disk Drive. Introduction - 8.1

Input / Ouput devices. I/O Chapter 8. Goals & Constraints. Measures of Performance. Anatomy of a Disk Drive. Introduction - 8.1 Introduction - 8.1 I/O Chapter 8 Disk Storage and Dependability 8.2 Buses and other connectors 8.4 I/O performance measures 8.6 Input / Ouput devices keyboard, mouse, printer, game controllers, hard drive,

More information

Write a technical report Present your results Write a workshop/conference paper (optional) Could be a real system, simulation and/or theoretical

Write a technical report Present your results Write a workshop/conference paper (optional) Could be a real system, simulation and/or theoretical Identify a problem Review approaches to the problem Propose a novel approach to the problem Define, design, prototype an implementation to evaluate your approach Could be a real system, simulation and/or

More information

Storing Data: Disks and Files

Storing Data: Disks and Files Storing Data: Disks and Files [R&G] Chapter 9 CS 4320 1 Data on External Storage Disks: Can retrieve random page at fixed cost But reading several consecutive pages is much cheaper than reading them in

More information

Data Storage - II: Efficient Usage & Errors

Data Storage - II: Efficient Usage & Errors Data Storage - II: Efficient Usage & Errors Week 10, Spring 2005 Updated by M. Naci Akkøk, 27.02.2004, 03.03.2005 based upon slides by Pål Halvorsen, 12.3.2002. Contains slides from: Hector Garcia-Molina

More information

Chapter 13. Disk Storage, Basic File Structures, and Hashing

Chapter 13. Disk Storage, Basic File Structures, and Hashing Chapter 13 Disk Storage, Basic File Structures, and Hashing Chapter Outline Disk Storage Devices Files of Records Operations on Files Unordered Files Ordered Files Hashed Files Dynamic and Extendible Hashing

More information

HARD DRIVE CHARACTERISTICS REFRESHER

HARD DRIVE CHARACTERISTICS REFRESHER The read/write head of a hard drive only detects changes in the magnetic polarity of the material passing beneath it, not the direction of the polarity. Writes are performed by sending current either one

More information

Today: Coda, xfs. Coda Overview

Today: Coda, xfs. Coda Overview Today: Coda, xfs Case Study: Coda File System Brief overview of other file systems xfs Log structured file systems HDFS Object Storage Systems CS677: Distributed OS Lecture 21, page 1 Coda Overview DFS

More information

Storage in Database Systems. CMPSCI 445 Fall 2010

Storage in Database Systems. CMPSCI 445 Fall 2010 Storage in Database Systems CMPSCI 445 Fall 2010 1 Storage Topics Architecture and Overview Disks Buffer management Files of records 2 DBMS Architecture Query Parser Query Rewriter Query Optimizer Query

More information

Fault Tolerance & Reliability CDA 5140. Chapter 3 RAID & Sample Commercial FT Systems

Fault Tolerance & Reliability CDA 5140. Chapter 3 RAID & Sample Commercial FT Systems Fault Tolerance & Reliability CDA 5140 Chapter 3 RAID & Sample Commercial FT Systems - basic concept in these, as with codes, is redundancy to allow system to continue operation even if some components

More information

Introduction. What is RAID? The Array and RAID Controller Concept. Click here to print this article. Re-Printed From SLCentral

Introduction. What is RAID? The Array and RAID Controller Concept. Click here to print this article. Re-Printed From SLCentral Click here to print this article. Re-Printed From SLCentral RAID: An In-Depth Guide To RAID Technology Author: Tom Solinap Date Posted: January 24th, 2001 URL: http://www.slcentral.com/articles/01/1/raid

More information

RAID Storage, Network File Systems, and DropBox

RAID Storage, Network File Systems, and DropBox RAID Storage, Network File Systems, and DropBox George Porter CSE 124 February 24, 2015 * Thanks to Dave Patterson and Hong Jiang Announcements Project 2 due by end of today Office hour today 2-3pm in

More information

Chapter 13 File and Database Systems

Chapter 13 File and Database Systems Chapter 13 File and Database Systems Outline 13.1 Introduction 13.2 Data Hierarchy 13.3 Files 13.4 File Systems 13.4.1 Directories 13.4. Metadata 13.4. Mounting 13.5 File Organization 13.6 File Allocation

More information

Chapter 13 File and Database Systems

Chapter 13 File and Database Systems Chapter 13 File and Database Systems Outline 13.1 Introduction 13.2 Data Hierarchy 13.3 Files 13.4 File Systems 13.4.1 Directories 13.4. Metadata 13.4. Mounting 13.5 File Organization 13.6 File Allocation

More information

Transactions and Reliability. Sarah Diesburg Operating Systems CS 3430

Transactions and Reliability. Sarah Diesburg Operating Systems CS 3430 Transactions and Reliability Sarah Diesburg Operating Systems CS 3430 Motivation File systems have lots of metadata: Free blocks, directories, file headers, indirect blocks Metadata is heavily cached for

More information

Operating Systems CSE 410, Spring 2004. File Management. Stephen Wagner Michigan State University

Operating Systems CSE 410, Spring 2004. File Management. Stephen Wagner Michigan State University Operating Systems CSE 410, Spring 2004 File Management Stephen Wagner Michigan State University File Management File management system has traditionally been considered part of the operating system. Applications

More information

RAID HARDWARE. On board SATA RAID controller. RAID drive caddy (hot swappable) SATA RAID controller card. Anne Watson 1

RAID HARDWARE. On board SATA RAID controller. RAID drive caddy (hot swappable) SATA RAID controller card. Anne Watson 1 RAID HARDWARE On board SATA RAID controller SATA RAID controller card RAID drive caddy (hot swappable) Anne Watson 1 RAID The word redundant means an unnecessary repetition. The word array means a lineup.

More information

Lecture 18: Reliable Storage

Lecture 18: Reliable Storage CS 422/522 Design & Implementation of Operating Systems Lecture 18: Reliable Storage Zhong Shao Dept. of Computer Science Yale University Acknowledgement: some slides are taken from previous versions of

More information

1 Storage Devices Summary

1 Storage Devices Summary Chapter 1 Storage Devices Summary Dependability is vital Suitable measures Latency how long to the first bit arrives Bandwidth/throughput how fast does stuff come through after the latency period Obvious

More information

Chapter 13 Disk Storage, Basic File Structures, and Hashing.

Chapter 13 Disk Storage, Basic File Structures, and Hashing. Chapter 13 Disk Storage, Basic File Structures, and Hashing. Copyright 2004 Pearson Education, Inc. Chapter Outline Disk Storage Devices Files of Records Operations on Files Unordered Files Ordered Files

More information

HP Smart Array Controllers and basic RAID performance factors

HP Smart Array Controllers and basic RAID performance factors Technical white paper HP Smart Array Controllers and basic RAID performance factors Technology brief Table of contents Abstract 2 Benefits of drive arrays 2 Factors that affect performance 2 HP Smart Array

More information

Lecture 5: GFS & HDFS! Claudia Hauff (Web Information Systems)! ti2736b-ewi@tudelft.nl

Lecture 5: GFS & HDFS! Claudia Hauff (Web Information Systems)! ti2736b-ewi@tudelft.nl Big Data Processing, 2014/15 Lecture 5: GFS & HDFS!! Claudia Hauff (Web Information Systems)! ti2736b-ewi@tudelft.nl 1 Course content Introduction Data streams 1 & 2 The MapReduce paradigm Looking behind

More information

PIONEER RESEARCH & DEVELOPMENT GROUP

PIONEER RESEARCH & DEVELOPMENT GROUP SURVEY ON RAID Aishwarya Airen 1, Aarsh Pandit 2, Anshul Sogani 3 1,2,3 A.I.T.R, Indore. Abstract RAID stands for Redundant Array of Independent Disk that is a concept which provides an efficient way for

More information

File-System Implementation

File-System Implementation File-System Implementation 11 CHAPTER In this chapter we discuss various methods for storing information on secondary storage. The basic issues are device directory, free space management, and space allocation

More information

Overview of I/O Performance and RAID in an RDBMS Environment. By: Edward Whalen Performance Tuning Corporation

Overview of I/O Performance and RAID in an RDBMS Environment. By: Edward Whalen Performance Tuning Corporation Overview of I/O Performance and RAID in an RDBMS Environment By: Edward Whalen Performance Tuning Corporation Abstract This paper covers the fundamentals of I/O topics and an overview of RAID levels commonly

More information

Chapter 10: Mass-Storage Systems

Chapter 10: Mass-Storage Systems Chapter 10: Mass-Storage Systems Physical structure of secondary storage devices and its effects on the uses of the devices Performance characteristics of mass-storage devices Disk scheduling algorithms

More information

File System & Device Drive. Overview of Mass Storage Structure. Moving head Disk Mechanism. HDD Pictures 11/13/2014. CS341: Operating System

File System & Device Drive. Overview of Mass Storage Structure. Moving head Disk Mechanism. HDD Pictures 11/13/2014. CS341: Operating System CS341: Operating System Lect 36: 1 st Nov 2014 Dr. A. Sahu Dept of Comp. Sc. & Engg. Indian Institute of Technology Guwahati File System & Device Drive Mass Storage Disk Structure Disk Arm Scheduling RAID

More information

Lecture 23: Multiprocessors

Lecture 23: Multiprocessors Lecture 23: Multiprocessors Today s topics: RAID Multiprocessor taxonomy Snooping-based cache coherence protocol 1 RAID 0 and RAID 1 RAID 0 has no additional redundancy (misnomer) it uses an array of disks

More information

Storing Data: Disks and Files

Storing Data: Disks and Files Storing : Disks and Files Chapter 7 base Management Systems, R. Ramakrishnan and J. Gehrke 1 Disks and Files DBMS stores information on ( hard ) disks. This has major implications for DBMS implementation!

More information

Storing Data: Disks and Files

Storing Data: Disks and Files Storing Data: Disks and Files Chapter 7 Yea, from the table of my memory I ll wipe away all trivial fond records. -- Shakespeare, Hamlet Database Management Systems, R. Ramakrishnan and J. Gehrke 1 Disks

More information

Storing Data: Disks and Files

Storing Data: Disks and Files Storing Data: Disks and Files Chapter 9 Comp 521 Files and Databases Fall 2010 1 Disks and Files DBMS stores information on ( hard ) disks. This has major implications for DBMS design! READ: transfer data

More information

Why multi-disk systems?

Why multi-disk systems? RAID Why multi-disk systems? A single storage device may not provide: enough storage capacity performance capacity reliability Why multi-disk systems? A single storage device may not provide: enough storage

More information

Chapter 11 I/O Management and Disk Scheduling

Chapter 11 I/O Management and Disk Scheduling Operatin g Systems: Internals and Design Principle s Chapter 11 I/O Management and Disk Scheduling Seventh Edition By William Stallings Operating Systems: Internals and Design Principles An artifact can

More information

Copyright 2007 Ramez Elmasri and Shamkant B. Navathe. Slide 13-1

Copyright 2007 Ramez Elmasri and Shamkant B. Navathe. Slide 13-1 Slide 13-1 Chapter 13 Disk Storage, Basic File Structures, and Hashing Chapter Outline Disk Storage Devices Files of Records Operations on Files Unordered Files Ordered Files Hashed Files Dynamic and Extendible

More information

Storage. The text highlighted in green in these slides contain external hyperlinks. 1 / 14

Storage. The text highlighted in green in these slides contain external hyperlinks. 1 / 14 Storage Compared to the performance parameters of the other components we have been studying, storage systems are much slower devices. Typical access times to rotating disk storage devices are in the millisecond

More information

Google File System. Web and scalability

Google File System. Web and scalability Google File System Web and scalability The web: - How big is the Web right now? No one knows. - Number of pages that are crawled: o 100,000 pages in 1994 o 8 million pages in 2005 - Crawlable pages might

More information

Non-Redundant (RAID Level 0)

Non-Redundant (RAID Level 0) There are many types of RAID and some of the important ones are introduced below: Non-Redundant (RAID Level 0) A non-redundant disk array, or RAID level 0, has the lowest cost of any RAID organization

More information

RAID. RAID 0 No redundancy ( AID?) Just stripe data over multiple disks But it does improve performance. Chapter 6 Storage and Other I/O Topics 29

RAID. RAID 0 No redundancy ( AID?) Just stripe data over multiple disks But it does improve performance. Chapter 6 Storage and Other I/O Topics 29 RAID Redundant Array of Inexpensive (Independent) Disks Use multiple smaller disks (c.f. one large disk) Parallelism improves performance Plus extra disk(s) for redundant data storage Provides fault tolerant

More information

technology brief RAID Levels March 1997 Introduction Characteristics of RAID Levels

technology brief RAID Levels March 1997 Introduction Characteristics of RAID Levels technology brief RAID Levels March 1997 Introduction RAID is an acronym for Redundant Array of Independent Disks (originally Redundant Array of Inexpensive Disks) coined in a 1987 University of California

More information

Hard Disk Drives and RAID

Hard Disk Drives and RAID Hard Disk Drives and RAID Janaka Harambearachchi (Engineer/Systems Development) INTERFACES FOR HDD A computer interfaces is what allows a computer to send and retrieve information for storage devices such

More information

Chapter 16 Distributed-File Systems

Chapter 16 Distributed-File Systems Chapter 16 Distributed-File Systems Background Naming and Transparency Remote File Access Stateful versus Stateless Service File Replication Example Systems 16.1 Background Distributed file system (DFS)

More information

COS 318: Operating Systems. Snapshot and NFS

COS 318: Operating Systems. Snapshot and NFS COS 318: Operating Systems Snapshot and NFS Andy Bavier Computer Science Department Princeton University http://www.cs.princeton.edu/courses/archive/fall10/cos318/ Topics Revisit Transactions and Logging

More information

Distributed File Systems. NFS Architecture (1)

Distributed File Systems. NFS Architecture (1) COP 6611 Advanced Operating System Distributed File Systems Chi Zhang czhang@cs.fiu.edu NFS Architecture (1) a) The remote access model. (like NFS) b) The upload/download model (like FTP) 2 1 NFS Architecture

More information

RAID: Redundant Arrays of Inexpensive Disks this discussion is based on the paper: on Management of Data (Chicago, IL), pp.109--116, 1988.

RAID: Redundant Arrays of Inexpensive Disks this discussion is based on the paper: on Management of Data (Chicago, IL), pp.109--116, 1988. : Redundant Arrays of Inexpensive Disks this discussion is based on the paper:» A Case for Redundant Arrays of Inexpensive Disks (),» David A Patterson, Garth Gibson, and Randy H Katz,» In Proceedings

More information

Data Storage - II: Efficient Usage & Errors. Contains slides from: Naci Akkök, Hector Garcia-Molina, Pål Halvorsen, Ketil Lund

Data Storage - II: Efficient Usage & Errors. Contains slides from: Naci Akkök, Hector Garcia-Molina, Pål Halvorsen, Ketil Lund Data Storage - II: Efficient Usage & Errors Contains slides from: Naci Akkök, Hector Garcia-Molina, Pål Halvorsen, Ketil Lund Overview Efficient storage usage Disk errors Error recovery INF3 7.3.26 Ellen

More information

Why disk arrays? CPUs improving faster than disks

Why disk arrays? CPUs improving faster than disks Why disk arrays? CPUs improving faster than disks - disks will increasingly be bottleneck New applications (audio/video) require big files (motivation for XFS) Disk arrays - make one logical disk out of

More information

The Google File System (GFS)

The Google File System (GFS) The Google File System (GFS) Google File System Example of clustered file system Basis of Hadoop s and Bigtable s underlying file system Many other implementations Design constraints Motivating application:

More information

Summer Student Project Report

Summer Student Project Report Summer Student Project Report Dimitris Kalimeris National and Kapodistrian University of Athens June September 2014 Abstract This report will outline two projects that were done as part of a three months

More information

Objectives and Functions

Objectives and Functions Objectives and Functions William Stallings Computer Organization and Architecture 6 th Edition Week 10 Operating System Support Convenience Making the computer easier to use Efficiency Allowing better

More information

Chapter 6 External Memory. Dr. Mohamed H. Al-Meer

Chapter 6 External Memory. Dr. Mohamed H. Al-Meer Chapter 6 External Memory Dr. Mohamed H. Al-Meer 6.1 Magnetic Disks Types of External Memory Magnetic Disks RAID Removable Optical CD ROM CD Recordable CD-R CD Re writable CD-RW DVD Magnetic Tape 2 Introduction

More information

RAID Overview 91.520

RAID Overview 91.520 RAID Overview 91.520 1 The Motivation for RAID Computing speeds double every 3 years Disk speeds can t keep up Data needs higher MTBF than any component in system IO Performance and Availability Issues!

More information

COS 318: Operating Systems. File Layout and Directories. Topics. File System Components. Steps to Open A File

COS 318: Operating Systems. File Layout and Directories. Topics. File System Components. Steps to Open A File Topics COS 318: Operating Systems File Layout and Directories File system structure Disk allocation and i-nodes Directory and link implementations Physical layout for performance 2 File System Components

More information

File Systems: Fundamentals

File Systems: Fundamentals Files What is a file? A named collection of related information recorded on secondary storage (e.g., disks) File Systems: Fundamentals File attributes Name, type, location, size, protection, creator, creation

More information

Web Email DNS Peer-to-peer systems (file sharing, CDNs, cycle sharing)

Web Email DNS Peer-to-peer systems (file sharing, CDNs, cycle sharing) 1 1 Distributed Systems What are distributed systems? How would you characterize them? Components of the system are located at networked computers Cooperate to provide some service No shared memory Communication

More information

Chapter 11: File System Implementation. Operating System Concepts with Java 8 th Edition

Chapter 11: File System Implementation. Operating System Concepts with Java 8 th Edition Chapter 11: File System Implementation 11.1 Silberschatz, Galvin and Gagne 2009 Chapter 11: File System Implementation File-System Structure File-System Implementation Directory Implementation Allocation

More information

Physical Data Organization

Physical Data Organization Physical Data Organization Database design using logical model of the database - appropriate level for users to focus on - user independence from implementation details Performance - other major factor

More information

CS161: Operating Systems

CS161: Operating Systems CS161: Operating Systems Matt Welsh mdw@eecs.harvard.edu Lecture 18: RAID April 19, 2007 2007 Matt Welsh Harvard University 1 RAID Redundant Arrays of Inexpensive Disks Invented in 1986-1987 by David Patterson

More information

Chapter 17: Distributed-File Systems. Operating System Concepts 8 th Edition,

Chapter 17: Distributed-File Systems. Operating System Concepts 8 th Edition, Chapter 17: Distributed-File Systems, Silberschatz, Galvin and Gagne 2009 Chapter 17 Distributed-File Systems Background Naming and Transparency Remote File Access Stateful versus Stateless Service File

More information

StorTrends RAID Considerations

StorTrends RAID Considerations StorTrends RAID Considerations MAN-RAID 04/29/2011 Copyright 1985-2011 American Megatrends, Inc. All rights reserved. American Megatrends, Inc. 5555 Oakbrook Parkway, Building 200 Norcross, GA 30093 Revision

More information

PARALLELS CLOUD STORAGE

PARALLELS CLOUD STORAGE PARALLELS CLOUD STORAGE Performance Benchmark Results 1 Table of Contents Executive Summary... Error! Bookmark not defined. Architecture Overview... 3 Key Features... 5 No Special Hardware Requirements...

More information

Distributed File System. MCSN N. Tonellotto Complements of Distributed Enabling Platforms

Distributed File System. MCSN N. Tonellotto Complements of Distributed Enabling Platforms Distributed File System 1 How do we get data to the workers? NAS Compute Nodes SAN 2 Distributed File System Don t move data to workers move workers to the data! Store data on the local disks of nodes

More information

Big data management with IBM General Parallel File System

Big data management with IBM General Parallel File System Big data management with IBM General Parallel File System Optimize storage management and boost your return on investment Highlights Handles the explosive growth of structured and unstructured data Offers

More information

We mean.network File System

We mean.network File System We mean.network File System Introduction: Remote File-systems When networking became widely available users wanting to share files had to log in across the net to a central machine This central machine

More information

CS 6290 I/O and Storage. Milos Prvulovic

CS 6290 I/O and Storage. Milos Prvulovic CS 6290 I/O and Storage Milos Prvulovic Storage Systems I/O performance (bandwidth, latency) Bandwidth improving, but not as fast as CPU Latency improving very slowly Consequently, by Amdahl s Law: fraction

More information

Disk Array Data Organizations and RAID

Disk Array Data Organizations and RAID Guest Lecture for 15-440 Disk Array Data Organizations and RAID October 2010, Greg Ganger 1 Plan for today Why have multiple disks? Storage capacity, performance capacity, reliability Load distribution

More information

PARALLEL I/O FOR HIGH PERFORMANCE COMPUTING

PARALLEL I/O FOR HIGH PERFORMANCE COMPUTING o. rof. Dr. eter Brezany arallele and Verteilte Datenbanksysteme 1 ARALLEL I/O FOR HIGH ERFORMANCE COMUTING Skriptum zur Vorlesung eter Brezany Institut für Scientific Computing Universität Wien E-Mail:

More information

Cloud Storage. Parallels. Performance Benchmark Results. White Paper. www.parallels.com

Cloud Storage. Parallels. Performance Benchmark Results. White Paper. www.parallels.com Parallels Cloud Storage White Paper Performance Benchmark Results www.parallels.com Table of Contents Executive Summary... 3 Architecture Overview... 3 Key Features... 4 No Special Hardware Requirements...

More information

High-Performance SSD-Based RAID Storage. Madhukar Gunjan Chakhaiyar Product Test Architect

High-Performance SSD-Based RAID Storage. Madhukar Gunjan Chakhaiyar Product Test Architect High-Performance SSD-Based RAID Storage Madhukar Gunjan Chakhaiyar Product Test Architect 1 Agenda HDD based RAID Performance-HDD based RAID Storage Dynamics driving to SSD based RAID Storage Evolution

More information

RAID. Storage-centric computing, cloud computing. Benefits:

RAID. Storage-centric computing, cloud computing. Benefits: RAID Storage-centric computing, cloud computing. Benefits: Improved reliability (via error correcting code, redundancy). Improved performance (via redundancy). Independent disks. RAID Level 0 Provides

More information

Reliable Adaptable Network RAM

Reliable Adaptable Network RAM Reliable Adaptable Network RAM Tia Newhall, Daniel Amato, Alexandr Pshenichkin Computer Science Department, Swarthmore College Swarthmore, PA 19081, USA Abstract We present reliability solutions for adaptable

More information

CHAPTER 4 RAID. Section Goals. Upon completion of this section you should be able to:

CHAPTER 4 RAID. Section Goals. Upon completion of this section you should be able to: HPTER 4 RI s it was originally proposed, the acronym RI stood for Redundant rray of Inexpensive isks. However, it has since come to be known as Redundant rray of Independent isks. RI was originally described

More information