CS 464/564 Introduction to Database Management System Instructor: Abdullah Mueen

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "CS 464/564 Introduction to Database Management System Instructor: Abdullah Mueen"

Transcription

1 CS 464/564 Introduction to Database Management System Instructor: Abdullah Mueen LECTURE 14: DATA STORAGE AND REPRESENTATION

2 Data Storage Memory Hierarchy Disks Fields, Records, Blocks Variable-length Data Modifying Records 2

3 Strawman Implementation Use UNIX file system to store relations, e.g. Students(name, id, dept) in file /usr/db/students One line per tuple, each component stored as character string, with # as a separator, e.g. tuple could be: Smith#123#CS Store schema in /usr/db/schema, e.g.: Students#name#STR#id#INT#dept#STR 3

4 What's Wrong? The storage of the tuples on disk is inflexible: if a student changes major from EE to ECON, entire file must be rewritten Search is very expensive (read entire relation) Query processing is "brute force" -- there are faster ways to do joins, etc. Data is not buffered between disk and main memory No concurrency control No reliability in case of a crash 4

5 Memory Hierarchy Cache 1ns main memory 1-10ns faster, smaller, more expensive secondary storage (disk) 10ms tertiary storage (tapes, CD-ROM) slower, larger, cheaper 5

6 Cache Memory Transfer a few bytes at a time between cache and main memory: instruction, integer, floating point, short string Processor operates on instruction and data in the cache Typical size: 1 Mbyte (2 20 bytes) Typical speed to/from main memory: 10 nanosec (1 nanosec = 10-9 sec) 6

7 Main Memory Typical size: 100 Mbytes to 10 Gbytes (1 Gbyte = 2 30 bytes) Typical access speed (to read or write): 10 to 100 nanosec At least 100 times larger than cache At least 10 times slower than cache 7

8 Secondary Storage Usually disk Divided logically into blocks, unit of transfer between main memory (called disk I/O) Typical size: 100 Gbytes-10TB Typical speed: 10 millisec (10-3 sec) At least 100 times larger than main memory Much slower than main memory and much much slower than cache: can execute several million instructions during one disk I/O 8

9 Tertiary Storage Tape(s) CD-ROM(s) At least 1000 times slower than secondary storage At least 50% cheaper than secondary storage 9

10 Volatile vs. Nonvolatile Storage is volatile if the data is lost when the power is gone Usually main memory is volatile Usually secondary and tertiary storage is nonvolatile Thus every change made to a database in main memory must be backed up on disk before it can be permanent. 10

11 Disks platters: each has two surfaces, each surface consists of tracks (concentric rings) disk heads spindle one head per surface, very close to surface, does the reading and writing 11

12 More on Disks orange ring is a track black squares are gaps, which don't hold data part of track between two gaps is a sector one or more sectors make a block 12

13 Disk Controller controls mechanical actuator that moves the heads in and out (radius, distance from spindle) one track from each surface at the same radius forms a cylinder selects a surface selects a sector (senses when that sector is under the corresponding head) transfers bits 13

14 Typical Values (old vs. newer) Rotation speed (old): 5400 rmp Rotation speed (2014): 7200 rpm, 1 rotation in 8.33 ms Number of platters (old): 5 Number of platters (2014): 8 Number of tracks/surface: 20,000 Number of tracks/surface (2008): 65,536 Number of sectors/track: 500 Number of sectors/track (2008): 256 Number of bytes/sector: thousands Number of bytes/sector (2008): thousands 14

15 Disk Latency for a Read Time between issuing command to read a block and when contents of block appear in main memory: time for processor and disk controller to process request, including resolving any contention (negligible) seek time: time to move heads to correct radius (0 to ~18 millisec) rotational latency: time until first sector of block is under the head (~9 millisec) transfer time: until all sectors of the block have passed under the head; depends on rotation speed and size of block 15

16 Disk Latency for Updates For a write: like reading plus verification (read back and compare) To modify a block: read it into main memory change it in main memory write it back to disk 16

17 Moral of the Story Disks accesses are orders of magnitude slower than accesses to main memory. They are unavoidable in large databases. Thus do everything possible to minimize them. Can lead to different algorithms. 17

18 Speeding Up Disk Accesses 1. Place blocks accessed together on same cylinder reduces seek time and rotational latency 2. Divide data among several disks head assemblies can move in parallel 3. Mirror a disk: make copies of it speeds up reads: get data from disk whose head is closest to desired block no effect on writes: write to all copies also helps with fault tolerance 18

19 Speeding up Disk Accesses 4. Be clever about order in which read and write requests are serviced, i.e., algorithm in OS or DBMS or disk controller Ex: elevator algorithm 5. Prefetch blocks to main memory in anticipation of future use (buffering) 19

20 Elevator Algorithm Disk head assembly sweeps in and out repeatedly When heads pass a cylinder with pending requests, they stop to do the request When reaching a point with no pending requests ahead, change direction Works well when there are many "independent" read and write requests, i.e., don't need to be done in a particular order, that are randomly distributed over the disk. 20

21 Prefetching Suppose you can predict order in which blocks will be requested from disk. Load them into main memory buffers before they are needed. Have flexibility to schedule the reads efficiently Can also delay writing buffered blocks if the buffers are not needed immediately 21

22 Disk Failures Intermittent failure: attempt to read or write a sector fails but a subsequent try succeeds Impossible to read sector Impossible to write a sector Disk crash: entire disk becomes unreadable 22

23 Coping with Intermittent Failures Use redundant bits in each sector Store checksums in the redundant bits After a read, check if checksums are correct; if not then try again After a write, can do a read and compare with value written, or be optimistic and just check the checksum of the read 23

24 Checksums Suppose we use one extra bit, a parity bit. if the number of 1's in the data bits is odd, then set the parity bit to 1, otherwise to 0 This is not foolproof: 101 and 110 both have even parity so checksum would be 0 for both Use n parity bits in the checksum: parity bit 1 stores parity of every n-th bit, starting with first bit, parity bit 2 stores parity of every n-th bit, starting with second bit, etc. Probability of missing an error is 1/2 n 24

25 Coping with Permanent Read/Write Errors Stable storage policy: Each "virtual" sector X is represented by two real sectors, X L and X R. To write value v to X: repeat {write v to X L, read from X L } until read's checksum is correct or exceed max # of tries do the same thing with X R if X L or X R is discovered to be bad, then must find a substitute 25

26 Handling Write Failures Suppose write(s) to X L all fail. Then old value is safe in X R. Suppose write(s) to X R all fail. Then new value is safe in X L. Assumption is that it is highly unlikely for two sectors to fail around the same time. 26

27 More on Stable Storage To read from X: repeatedly read X L until checksum is good or exceed max # tries if read of X L failed then repeatedly read X R until checksum is good or exceed max # tries Handles permanent read failures, unless both X L and X R fail about the same time (unlikely) 27

28 Coping with Disk Crashes "Mean time to failure" of a disk is length of time by which 50% of such disks will have had a head crash Goal is to have a much longer "mean time to data loss" for your system Key idea: use redundancy Discuss three such approaches next 28

29 Mirroring (RAID Level 1) Keep another copy of each disk: write to both, read from one. Only way data can be lost is if second disk crashes while first is being repaired. If mean time to crash of a single disk is 10 years and it takes 3 hours to repair a disk, then mean time to data loss is 146,000 years. 29

30 Parity Blocks (RAID Level 4) Drawback of previous scheme is that you need double the number of disks. Instead use one spare disk no matter how many data disks you have. Block i of the spare disk contains the parity checks for block i of all the data disks. If spare disk fails, get a new spare. Image source: wikipedia If a data disk fails, recompute its data from the other data disks and the spare. 30

31 RAID Level 5 Drawback of previous scheme is that spare disk is a bottleneck. Instead, let each data disk also serve as the spare disk for some blocks. All these assume only one crash at a time. RAID Level 6 uses error-correcting codes to be able to handle multiple crashes. Image source: wikipedia 31

32 Data Representation Attributes are represented by sequences of bytes, called fields Tuples are represented by collections of fields, called records Relations are represented by collections of records, called files Files are stored in blocks, using specialized data structures to support efficient modification and querying 32

33 Representing SQL Data Types integers and reals: built-in CHAR(n): array of n bytes VARCHAR(n): array of n+1 bytes (extra byte is either string length or null char) dates and times: fixed length strings etc. 33

34 Representing Tuples For now, assume all attributes (fields) are fixed length. Concatenate the fields Store the offset of each field in schema name CHAR(30) 30 bytes address VARCHAR(255) 256 bytes gender CHAR(1) 1 byte birthdate DATE 10 bytes 34

35 More on Tuples Due to hardware considerations, certain types of data need to start at addresses that are multiples of 4 or 8 Previous example becomes: name CHAR(30) 30 bytes + 2 address VARCHAR(255) 256 bytes gender CHAR(1) 1 byte + 3 birthdate DATE 10 bytes

36 Record Headers Often it is convenient to keep some "header" information in each record: a pointer to schema information (attributes/fields, types, their order in the tuple, constraints) length of the record/tuple timestamp of last modification To Schema Length Timestamp name CHAR(30) 30 bytes address VARCHAR(255) 256 bytes gender CHAR(1) 1 byte birthdate DATE 10 bytes 36

37 Packing Records into Blocks Start with block header: timestamp of last modification/access offset of each record in the block, etc. Follow with sequence of records May end with some unused space header block 1 block 2 block n-1 block n 37

38 Representing Addresses Often addresses (pointers) are part of records: the application data in object-oriented databases as part of indexes and other data structures supporting the DBMS Every data item (block, record, etc.) has two addresses: database address: address on the disk (typically 8-16 bytes) memory address, if the item is in virtual memory (typically 4 bytes) 38

39 Translation Table Provides mapping from database addresses to memory addresses for all blocks currently in memory Later we'll discuss how to implement it 39

40 Pointer Swizzling When a block is moved from disk into main memory, change all the disk addresses that point to items in this block into main memory addresses. Need a bit for each address to indicate if it is a disk address or a memory address. Why? Faster to follow memory pointers (only uses a single machine instruction). 40

41 Example of Swizzling Disk Main Memory read into main memory Block 1 Block 2 41

42 Swizzling Policies Automatic swizzling: as soon as block is brought into memory, swizzle all relevant pointers Swizzling on demand: only swizzle a pointer if and when it is actually followed No swizzling Programmer control 42

43 Automatic Swizzling Locating all pointers within a block: refer to the schema, which will indicate where addresses are in the records for index structures, pointers are at known locations Update translation table with memory addresses of items in the block Update pointers in the block (in memory) with memory addresses, when possible, as obtained from translation table 43

44 Unswizzling When a block is moved from memory back to disk, all pointers must go back to database (disk) addresses Use translation table again Important to have an efficient data structure for the translation table 44

45 Pinned Records and Blocks A block in memory is pinned if it cannot be safely written back to disk Indicate with a bit in the block header Reasons for pinning: related to failure recovery (more later) because of pointer swizzling If block B1 has swizzled pointer to an item in block B2, then B2 is pinned. 45

46 Unpinning a Block Consider each item in the block to be unpinned Keep in the translation table the places in memory holding swizzled pointers to that item (e.g., with a linked list) Unswizzle those pointers (i.e., use translation table to replace the memory addresses with database (disk) addresses 46

47 Variable Length Data Data items with varying size (e.g., if maximum size of a field is large but most of the time the values are small) Variable-format records (e.g., NULLs method for representing a hierarchy of entity sets as relations) Records that do not fit in a block (e.g., an MPEG of a movie) 47

48 Variable-Length Fields Store the fixed-length fields before the variable-length fields in each record Keep in the record header record length pointers to the beginnings of all the variable-length fields Book discusses variations on this idea 48

49 Variable-Format Records Represent by a sequence of tagged fields Each tagged field contains name type length, if not deducible from the type value 49

50 Splitting Records Across Blocks Called spanned records Useful when record size exceeds block size putting an integral number of records in a block wastes a lot of the block (e.g., record size is 51% of block size) Each record or fragment header contains bit indicating if it is a fragment if fragment then pointers to previous and next fragments of the record (i.e., a linked list) 50

51 Record Modification Modifications to records: insert delete update issues even with fixed-length records and fields even more involved with variable-length data 51

52 Inserting New Records If records need not be any particular order, then just find a block with enough empty space Later we'll see how to keep track of all the tuples of a given relation But what if blocks should be kept in a certain order, such as sorted on primary key? 52

53 Insertion in Order header unused record 4 record 3 record 2 record 1 If there is space in the block, then add the record (going right to left), add a pointer to it (going left to right) and rearrange the pointers as needed. 53

54 What if Block is Full? Records are stored in several blocks, in sorted order One approach: keep a linked list of "overflow" blocks for each block in the main sequence Another approach: slide records down to another block, leave forwarding addresses in old block, slide records on both old and new blocks 54

55 Deleting Records Try to reclaim space made available after a record is deleted If using an offset table, then rearrange the records to fill in any hole that is left behind and adjust the pointers Additional mechanisms are based on keeping a linked list of available space and compacting when possible 55

56 Tombstones What about pointers to deleted records? We place a tombstone in place of each deleted record Tombstone is permanent Issue of where to place the tombstone Keep a tombstone bit in each record header: if this is a tombstone, then no need to store additional data 56

57 Updating Records For fixed-length records, there is no effect on the storage system For variable-length records: if length increases, like insertion if length decreases, like deletion except tombstones are not necessary 57

Database 2 Lecture I. Alessandro Artale

Database 2 Lecture I. Alessandro Artale Free University of Bolzano Database 2. Lecture I, 2003/2004 A.Artale (1) Database 2 Lecture I Alessandro Artale Faculty of Computer Science Free University of Bolzano Room: 221 artale@inf.unibz.it http://www.inf.unibz.it/

More information

System Architecture. CS143: Disks and Files. Magnetic disk vs SSD. Structure of a Platter CPU. Disk Controller...

System Architecture. CS143: Disks and Files. Magnetic disk vs SSD. Structure of a Platter CPU. Disk Controller... System Architecture CS143: Disks and Files CPU Word (1B 64B) ~ 10 GB/sec Main Memory System Bus Disk Controller... Block (512B 50KB) ~ 100 MB/sec Disk 1 2 Magnetic disk vs SSD Magnetic Disk Stores data

More information

Storage and File Structure

Storage and File Structure Storage and File Structure Chapter 10: Storage and File Structure Overview of Physical Storage Media Magnetic Disks RAID Tertiary Storage Storage Access File Organization Organization of Records in Files

More information

Storing Data: Disks and Files. Disks and Files. Why Not Store Everything in Main Memory? Chapter 7

Storing Data: Disks and Files. Disks and Files. Why Not Store Everything in Main Memory? Chapter 7 Storing : Disks and Files Chapter 7 Yea, from the table of my memory I ll wipe away all trivial fond records. -- Shakespeare, Hamlet base Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Disks and

More information

Chapter 2 Data Storage

Chapter 2 Data Storage Chapter 2 22 CHAPTER 2. DATA STORAGE 2.1. THE MEMORY HIERARCHY 23 26 CHAPTER 2. DATA STORAGE main memory, yet is essentially random-access, with relatively small differences Figure 2.4: A typical

More information

Physical Data Organization

Physical Data Organization Physical Data Organization Database design using logical model of the database - appropriate level for users to focus on - user independence from implementation details Performance - other major factor

More information

6. Storage and File Structures

6. Storage and File Structures ECS-165A WQ 11 110 6. Storage and File Structures Goals Understand the basic concepts underlying different storage media, buffer management, files structures, and organization of records in files. Contents

More information

Price/performance Modern Memory Hierarchy

Price/performance Modern Memory Hierarchy Lecture 21: Storage Administration Take QUIZ 15 over P&H 6.1-4, 6.8-9 before 11:59pm today Project: Cache Simulator, Due April 29, 2010 NEW OFFICE HOUR TIME: Tuesday 1-2, McKinley Last Time Exam discussion

More information

Principles of Database Management Systems. Overview. Principles of Data Layout. Topic for today. "Executive Summary": here.

Principles of Database Management Systems. Overview. Principles of Data Layout. Topic for today. Executive Summary: here. Topic for today Principles of Database Management Systems Pekka Kilpeläinen (after Stanford CS245 slide originals by Hector Garcia-Molina, Jeff Ullman and Jennifer Widom) How to represent data on disk

More information

Outline. Principles of Database Management Systems. Memory Hierarchy: Capacities and access times. CPU vs. Disk Speed ... ...

Outline. Principles of Database Management Systems. Memory Hierarchy: Capacities and access times. CPU vs. Disk Speed ... ... Outline Principles of Database Management Systems Pekka Kilpeläinen (after Stanford CS245 slide originals by Hector Garcia-Molina, Jeff Ullman and Jennifer Widom) Hardware: Disks Access Times Example -

More information

Copyright 2007 Ramez Elmasri and Shamkant B. Navathe. Slide 13-1

Copyright 2007 Ramez Elmasri and Shamkant B. Navathe. Slide 13-1 Slide 13-1 Chapter 13 Disk Storage, Basic File Structures, and Hashing Chapter Outline Disk Storage Devices Files of Records Operations on Files Unordered Files Ordered Files Hashed Files Dynamic and Extendible

More information

Storing Data: Disks and Files

Storing Data: Disks and Files Storing Data: Disks and Files (From Chapter 9 of textbook) Storing and Retrieving Data Database Management Systems need to: Store large volumes of data Store data reliably (so that data is not lost!) Retrieve

More information

Data Storage - I: Memory Hierarchies & Disks

Data Storage - I: Memory Hierarchies & Disks Data Storage - I: Memory Hierarchies & Disks W7-C, Spring 2005 Updated by M. Naci Akkøk, 27.02.2004 and 23.02.2005, based upon slides by Pål Halvorsen, 11.3.2002. Contains slides from: Hector Garcia-Molina,

More information

Storage in Database Systems. CMPSCI 445 Fall 2010

Storage in Database Systems. CMPSCI 445 Fall 2010 Storage in Database Systems CMPSCI 445 Fall 2010 1 Storage Topics Architecture and Overview Disks Buffer management Files of records 2 DBMS Architecture Query Parser Query Rewriter Query Optimizer Query

More information

Chapter 13. Disk Storage, Basic File Structures, and Hashing

Chapter 13. Disk Storage, Basic File Structures, and Hashing Chapter 13 Disk Storage, Basic File Structures, and Hashing Chapter Outline Disk Storage Devices Files of Records Operations on Files Unordered Files Ordered Files Hashed Files Dynamic and Extendible Hashing

More information

Chapter 13 Disk Storage, Basic File Structures, and Hashing.

Chapter 13 Disk Storage, Basic File Structures, and Hashing. Chapter 13 Disk Storage, Basic File Structures, and Hashing. Copyright 2004 Pearson Education, Inc. Chapter Outline Disk Storage Devices Files of Records Operations on Files Unordered Files Ordered Files

More information

Record Storage and Primary File Organization

Record Storage and Primary File Organization Record Storage and Primary File Organization 1 C H A P T E R 4 Contents Introduction Secondary Storage Devices Buffering of Blocks Placing File Records on Disk Operations on Files Files of Unordered Records

More information

Disks and RAID. Profs. Bracy and Van Renesse. based on slides by Prof. Sirer

Disks and RAID. Profs. Bracy and Van Renesse. based on slides by Prof. Sirer Disks and RAID Profs. Bracy and Van Renesse based on slides by Prof. Sirer 50 Years Old! 13th September 1956 The IBM RAMAC 350 Stored less than 5 MByte Reading from a Disk Must specify: cylinder # (distance

More information

Storing Data: Disks and Files

Storing Data: Disks and Files Storing Data: Disks and Files Chapter 9 Comp 521 Files and Databases Fall 2010 1 Disks and Files DBMS stores information on ( hard ) disks. This has major implications for DBMS design! READ: transfer data

More information

Chapter 13. Chapter Outline. Disk Storage, Basic File Structures, and Hashing

Chapter 13. Chapter Outline. Disk Storage, Basic File Structures, and Hashing Chapter 13 Disk Storage, Basic File Structures, and Hashing Copyright 2007 Ramez Elmasri and Shamkant B. Navathe Chapter Outline Disk Storage Devices Files of Records Operations on Files Unordered Files

More information

Data Storage - II: Efficient Usage & Errors

Data Storage - II: Efficient Usage & Errors Data Storage - II: Efficient Usage & Errors Week 10, Spring 2005 Updated by M. Naci Akkøk, 27.02.2004, 03.03.2005 based upon slides by Pål Halvorsen, 12.3.2002. Contains slides from: Hector Garcia-Molina

More information

INTRODUCTION The collection of data that makes up a computerized database must be stored physically on some computer storage medium.

INTRODUCTION The collection of data that makes up a computerized database must be stored physically on some computer storage medium. Chapter 4: Record Storage and Primary File Organization 1 Record Storage and Primary File Organization INTRODUCTION The collection of data that makes up a computerized database must be stored physically

More information

File System & Device Drive. Overview of Mass Storage Structure. Moving head Disk Mechanism. HDD Pictures 11/13/2014. CS341: Operating System

File System & Device Drive. Overview of Mass Storage Structure. Moving head Disk Mechanism. HDD Pictures 11/13/2014. CS341: Operating System CS341: Operating System Lect 36: 1 st Nov 2014 Dr. A. Sahu Dept of Comp. Sc. & Engg. Indian Institute of Technology Guwahati File System & Device Drive Mass Storage Disk Structure Disk Arm Scheduling RAID

More information

University of Dublin Trinity College. Storage Hardware. Owen.Conlan@cs.tcd.ie

University of Dublin Trinity College. Storage Hardware. Owen.Conlan@cs.tcd.ie University of Dublin Trinity College Storage Hardware Owen.Conlan@cs.tcd.ie Hardware Issues Hard Disk/SSD CPU Cache Main Memory CD ROM/RW DVD ROM/RW Tapes Primary Storage Floppy Disk/ Memory Stick Secondary

More information

Operating Systems CSE 410, Spring 2004. File Management. Stephen Wagner Michigan State University

Operating Systems CSE 410, Spring 2004. File Management. Stephen Wagner Michigan State University Operating Systems CSE 410, Spring 2004 File Management Stephen Wagner Michigan State University File Management File management system has traditionally been considered part of the operating system. Applications

More information

Outline. CS 245: Database System Principles. Notes 02: Hardware. Hardware DBMS ... ... Data Storage

Outline. CS 245: Database System Principles. Notes 02: Hardware. Hardware DBMS ... ... Data Storage CS 245: Database System Principles Notes 02: Hardware Hector Garcia-Molina Outline Hardware: Disks Access Times Solid State Drives Optimizations Other Topics: Storage costs Using secondary storage Disk

More information

Sistemas Operativos: Input/Output Disks

Sistemas Operativos: Input/Output Disks Sistemas Operativos: Input/Output Disks Pedro F. Souto (pfs@fe.up.pt) April 28, 2012 Topics Magnetic Disks RAID Solid State Disks Topics Magnetic Disks RAID Solid State Disks Magnetic Disk Construction

More information

COSC 6374 Parallel Computation. Parallel I/O (I) I/O basics. Concept of a clusters

COSC 6374 Parallel Computation. Parallel I/O (I) I/O basics. Concept of a clusters COSC 6374 Parallel I/O (I) I/O basics Fall 2012 Concept of a clusters Processor 1 local disks Compute node message passing network administrative network Memory Processor 2 Network card 1 Network card

More information

Chapter 10: Storage and File Structure

Chapter 10: Storage and File Structure Chapter 10: Storage and File Structure Overview of Physical Storage Media Magnetic Disks RAID Tertiary Storage Storage Access File Organization Organization of Records in Files Data-Dictionary Storage

More information

William Stallings Computer Organization and Architecture 7 th Edition. Chapter 6 External Memory

William Stallings Computer Organization and Architecture 7 th Edition. Chapter 6 External Memory William Stallings Computer Organization and Architecture 7 th Edition Chapter 6 External Memory Types of External Memory Magnetic Disk RAID Removable Optical CD-ROM CD-Recordable (CD-R) CD-R/W DVD Magnetic

More information

1 Storage Devices Summary

1 Storage Devices Summary Chapter 1 Storage Devices Summary Dependability is vital Suitable measures Latency how long to the first bit arrives Bandwidth/throughput how fast does stuff come through after the latency period Obvious

More information

Organization of Records in Blocks

Organization of Records in Blocks Organization of Records in Blocks Read Sec. 4.2 Riguzzi et al. Sistemi Informativi Slides derived from those by Hector Garcia-Molina 1 Topic How to lay out records on blocks 2 To represent: Integer (short):

More information

Chapter 10: Mass-Storage Systems

Chapter 10: Mass-Storage Systems Chapter 10: Mass-Storage Systems Physical structure of secondary storage devices and its effects on the uses of the devices Performance characteristics of mass-storage devices Disk scheduling algorithms

More information

COS 318: Operating Systems. Storage Devices. Kai Li Computer Science Department Princeton University. (http://www.cs.princeton.edu/courses/cos318/)

COS 318: Operating Systems. Storage Devices. Kai Li Computer Science Department Princeton University. (http://www.cs.princeton.edu/courses/cos318/) COS 318: Operating Systems Storage Devices Kai Li Computer Science Department Princeton University (http://www.cs.princeton.edu/courses/cos318/) Today s Topics Magnetic disks Magnetic disk performance

More information

HARD DRIVE CHARACTERISTICS REFRESHER

HARD DRIVE CHARACTERISTICS REFRESHER The read/write head of a hard drive only detects changes in the magnetic polarity of the material passing beneath it, not the direction of the polarity. Writes are performed by sending current either one

More information

Chapter 6. 6.1 Introduction. Storage and Other I/O Topics. p. 570( 頁 585) Fig. 6.1. I/O devices can be characterized by. I/O bus connections

Chapter 6. 6.1 Introduction. Storage and Other I/O Topics. p. 570( 頁 585) Fig. 6.1. I/O devices can be characterized by. I/O bus connections Chapter 6 Storage and Other I/O Topics 6.1 Introduction I/O devices can be characterized by Behavior: input, output, storage Partner: human or machine Data rate: bytes/sec, transfers/sec I/O bus connections

More information

COS 318: Operating Systems. Storage Devices. Kai Li and Andy Bavier Computer Science Department Princeton University

COS 318: Operating Systems. Storage Devices. Kai Li and Andy Bavier Computer Science Department Princeton University COS 318: Operating Systems Storage Devices Kai Li and Andy Bavier Computer Science Department Princeton University http://www.cs.princeton.edu/courses/archive/fall13/cos318/ Today s Topics! Magnetic disks!

More information

Lecture 16: Storage Devices

Lecture 16: Storage Devices CS 422/522 Design & Implementation of Operating Systems Lecture 16: Storage Devices Zhong Shao Dept. of Computer Science Yale University Acknowledgement: some slides are taken from previous versions of

More information

Disk Storage & Dependability

Disk Storage & Dependability Disk Storage & Dependability Computer Organization Architectures for Embedded Computing Wednesday 19 November 14 Many slides adapted from: Computer Organization and Design, Patterson & Hennessy 4th Edition,

More information

Secondary Storage. Any modern computer system will incorporate (at least) two levels of storage: magnetic disk/optical devices/tape systems

Secondary Storage. Any modern computer system will incorporate (at least) two levels of storage: magnetic disk/optical devices/tape systems 1 Any modern computer system will incorporate (at least) two levels of storage: primary storage: typical capacity cost per MB $3. typical access time burst transfer rate?? secondary storage: typical capacity

More information

1 File Management. 1.1 Naming. COMP 242 Class Notes Section 6: File Management

1 File Management. 1.1 Naming. COMP 242 Class Notes Section 6: File Management COMP 242 Class Notes Section 6: File Management 1 File Management We shall now examine how an operating system provides file management. We shall define a file to be a collection of permanent data with

More information

Input / Ouput devices. I/O Chapter 8. Goals & Constraints. Measures of Performance. Anatomy of a Disk Drive. Introduction - 8.1

Input / Ouput devices. I/O Chapter 8. Goals & Constraints. Measures of Performance. Anatomy of a Disk Drive. Introduction - 8.1 Introduction - 8.1 I/O Chapter 8 Disk Storage and Dependability 8.2 Buses and other connectors 8.4 I/O performance measures 8.6 Input / Ouput devices keyboard, mouse, printer, game controllers, hard drive,

More information

File System Management

File System Management Lecture 7: Storage Management File System Management Contents Non volatile memory Tape, HDD, SSD Files & File System Interface Directories & their Organization File System Implementation Disk Space Allocation

More information

CS 6290 I/O and Storage. Milos Prvulovic

CS 6290 I/O and Storage. Milos Prvulovic CS 6290 I/O and Storage Milos Prvulovic Storage Systems I/O performance (bandwidth, latency) Bandwidth improving, but not as fast as CPU Latency improving very slowly Consequently, by Amdahl s Law: fraction

More information

Introduction to I/O and Disk Management

Introduction to I/O and Disk Management Introduction to I/O and Disk Management 1 Secondary Storage Management Disks just like memory, only different Why have disks? Memory is small. Disks are large. Short term storage for memory contents (e.g.,

More information

Chapter 6: Physical Database Design and Performance. Database Development Process. Physical Design Process. Physical Database Design

Chapter 6: Physical Database Design and Performance. Database Development Process. Physical Design Process. Physical Database Design Chapter 6: Physical Database Design and Performance Modern Database Management 6 th Edition Jeffrey A. Hoffer, Mary B. Prescott, Fred R. McFadden Robert C. Nickerson ISYS 464 Spring 2003 Topic 23 Database

More information

Chapter 9: Peripheral Devices: Magnetic Disks

Chapter 9: Peripheral Devices: Magnetic Disks Chapter 9: Peripheral Devices: Magnetic Disks Basic Disk Operation Performance Parameters and History of Improvement Example disks RAID (Redundant Arrays of Inexpensive Disks) Improving Reliability Improving

More information

RAID. RAID 0 No redundancy ( AID?) Just stripe data over multiple disks But it does improve performance. Chapter 6 Storage and Other I/O Topics 29

RAID. RAID 0 No redundancy ( AID?) Just stripe data over multiple disks But it does improve performance. Chapter 6 Storage and Other I/O Topics 29 RAID Redundant Array of Inexpensive (Independent) Disks Use multiple smaller disks (c.f. one large disk) Parallelism improves performance Plus extra disk(s) for redundant data storage Provides fault tolerant

More information

Chapter 11: Storage and File Structure. Classification of Physical Storage Media

Chapter 11: Storage and File Structure. Classification of Physical Storage Media Chapter 11: Storage and File Structure! Overview of Physical Storage Media! Magnetic Disks! RAID! Tertiary Storage! Storage Access! File Organization! Organization of Records in Files! Data-Dictionary

More information

Database Management Systems

Database Management Systems 4411 Database Management Systems Acknowledgements and copyrights: these slides are a result of combination of notes and slides with contributions from: Michael Kiffer, Arthur Bernstein, Philip Lewis, Anestis

More information

Classification of Physical Storage Media. Chapter 11: Storage and File Structure. Physical Storage Media (Cont.) Physical Storage Media

Classification of Physical Storage Media. Chapter 11: Storage and File Structure. Physical Storage Media (Cont.) Physical Storage Media Chapter 11: Storage and File Structure Classification of Physical Storage Media! Overview of Physical Storage Media! Magnetic Disks! RAID! Tertiary Storage! Storage Access! File Organization! Organization

More information

CS 153 Design of Operating Systems Spring 2015

CS 153 Design of Operating Systems Spring 2015 CS 153 Design of Operating Systems Spring 2015 Lecture 22: File system optimizations Physical Disk Structure Disk components Platters Surfaces Tracks Arm Track Sector Surface Sectors Cylinders Arm Heads

More information

Chapter 8: Structures for Files. Truong Quynh Chi tqchi@cse.hcmut.edu.vn. Spring- 2013

Chapter 8: Structures for Files. Truong Quynh Chi tqchi@cse.hcmut.edu.vn. Spring- 2013 Chapter 8: Data Storage, Indexing Structures for Files Truong Quynh Chi tqchi@cse.hcmut.edu.vn Spring- 2013 Overview of Database Design Process 2 Outline Data Storage Disk Storage Devices Files of Records

More information

Chapter 2: Computer-System Structures. Computer System Operation Storage Structure Storage Hierarchy Hardware Protection General System Architecture

Chapter 2: Computer-System Structures. Computer System Operation Storage Structure Storage Hierarchy Hardware Protection General System Architecture Chapter 2: Computer-System Structures Computer System Operation Storage Structure Storage Hierarchy Hardware Protection General System Architecture Operating System Concepts 2.1 Computer-System Architecture

More information

COSC 6374 Parallel Computation. Parallel I/O (I) I/O basics. Concept of a clusters

COSC 6374 Parallel Computation. Parallel I/O (I) I/O basics. Concept of a clusters COSC 6374 Parallel Computation Parallel I/O (I) I/O basics Spring 2008 Concept of a clusters Processor 1 local disks Compute node message passing network administrative network Memory Processor 2 Network

More information

Lecture 1: Data Storage & Index

Lecture 1: Data Storage & Index Lecture 1: Data Storage & Index R&G Chapter 8-11 Concurrency control Query Execution and Optimization Relational Operators File & Access Methods Buffer Management Disk Space Management Recovery Manager

More information

The Classical Architecture. Storage 1 / 36

The Classical Architecture. Storage 1 / 36 1 / 36 The Problem Application Data? Filesystem Logical Drive Physical Drive 2 / 36 Requirements There are different classes of requirements: Data Independence application is shielded from physical storage

More information

File Management. Chapter 12

File Management. Chapter 12 Chapter 12 File Management File is the basic element of most of the applications, since the input to an application, as well as its output, is usually a file. They also typically outlive the execution

More information

Systems Infrastructure for Data Science. Web Science Group Uni Freiburg WS 2014/15

Systems Infrastructure for Data Science. Web Science Group Uni Freiburg WS 2014/15 Systems Infrastructure for Data Science Web Science Group Uni Freiburg WS 2014/15 Lecture I: Storage Storage Part I of this course Uni Freiburg, WS 2014/15 Systems Infrastructure for Data Science 3 The

More information

CHAPTER 13: DISK STORAGE, BASIC FILE STRUCTURES, AND HASHING

CHAPTER 13: DISK STORAGE, BASIC FILE STRUCTURES, AND HASHING Chapter 13: Disk Storage, Basic File Structures, and Hashing 1 CHAPTER 13: DISK STORAGE, BASIC FILE STRUCTURES, AND HASHING Answers to Selected Exercises 13.23 Consider a disk with the following characteristics

More information

Chapter 11 I/O Management and Disk Scheduling

Chapter 11 I/O Management and Disk Scheduling Operating Systems: Internals and Design Principles, 6/E William Stallings Chapter 11 I/O Management and Disk Scheduling Dave Bremer Otago Polytechnic, NZ 2008, Prentice Hall I/O Devices Roadmap Organization

More information

Introduction Disks RAID Tertiary storage. Mass Storage. CMSC 412, University of Maryland. Guest lecturer: David Hovemeyer.

Introduction Disks RAID Tertiary storage. Mass Storage. CMSC 412, University of Maryland. Guest lecturer: David Hovemeyer. Guest lecturer: David Hovemeyer November 15, 2004 The memory hierarchy Red = Level Access time Capacity Features Registers nanoseconds 100s of bytes fixed Cache nanoseconds 1-2 MB fixed RAM nanoseconds

More information

Disk Storage and Basic File Structure

Disk Storage and Basic File Structure 19 Disk Storage and Basic File Structure Marios Hadjieleftheriou AT&T Labs Apostolos N. Papadopoulos Aristotle University Donghui Zhang Northeastern University 19.1 Introduction............................................

More information

Devices and Device Controllers

Devices and Device Controllers I/O 1 Devices and Device Controllers network interface graphics adapter secondary storage (disks, tape) and storage controllers serial (e.g., mouse, keyboard) sound co-processors... I/O 2 Bus Architecture

More information

Chapter 12 File Management

Chapter 12 File Management Operating Systems: Internals and Design Principles, 6/E William Stallings Chapter 12 File Management Dave Bremer Otago Polytechnic, N.Z. 2008, Prentice Hall Roadmap Overview File organisation and Access

More information

Chapter 13 File and Database Systems

Chapter 13 File and Database Systems Chapter 13 File and Database Systems Outline 13.1 Introduction 13.2 Data Hierarchy 13.3 Files 13.4 File Systems 13.4.1 Directories 13.4. Metadata 13.4. Mounting 13.5 File Organization 13.6 File Allocation

More information

Chapter 13 File and Database Systems

Chapter 13 File and Database Systems Chapter 13 File and Database Systems Outline 13.1 Introduction 13.2 Data Hierarchy 13.3 Files 13.4 File Systems 13.4.1 Directories 13.4. Metadata 13.4. Mounting 13.5 File Organization 13.6 File Allocation

More information

Chapter 12 File Management. Roadmap

Chapter 12 File Management. Roadmap Operating Systems: Internals and Design Principles, 6/E William Stallings Chapter 12 File Management Dave Bremer Otago Polytechnic, N.Z. 2008, Prentice Hall Overview Roadmap File organisation and Access

More information

Operating Systems. RAID Redundant Array of Independent Disks. Submitted by Ankur Niyogi 2003EE20367

Operating Systems. RAID Redundant Array of Independent Disks. Submitted by Ankur Niyogi 2003EE20367 Operating Systems RAID Redundant Array of Independent Disks Submitted by Ankur Niyogi 2003EE20367 YOUR DATA IS LOST@#!! Do we have backups of all our data???? - The stuff we cannot afford to lose?? How

More information

Operating Systems: Internals and Design Principles. Chapter 12 File Management Seventh Edition By William Stallings

Operating Systems: Internals and Design Principles. Chapter 12 File Management Seventh Edition By William Stallings Operating Systems: Internals and Design Principles Chapter 12 File Management Seventh Edition By William Stallings Operating Systems: Internals and Design Principles If there is one singular characteristic

More information

CS161: Operating Systems

CS161: Operating Systems CS161: Operating Systems Matt Welsh mdw@eecs.harvard.edu Lecture 18: RAID April 19, 2007 2007 Matt Welsh Harvard University 1 RAID Redundant Arrays of Inexpensive Disks Invented in 1986-1987 by David Patterson

More information

Chapter 1 File Organization 1.0 OBJECTIVES 1.1 INTRODUCTION 1.2 STORAGE DEVICES CHARACTERISTICS

Chapter 1 File Organization 1.0 OBJECTIVES 1.1 INTRODUCTION 1.2 STORAGE DEVICES CHARACTERISTICS Chapter 1 File Organization 1.0 Objectives 1.1 Introduction 1.2 Storage Devices Characteristics 1.3 File Organization 1.3.1 Sequential Files 1.3.2 Indexing and Methods of Indexing 1.3.3 Hash Files 1.4

More information

Chapter 12 File Management

Chapter 12 File Management Operating Systems: Internals and Design Principles Chapter 12 File Management Eighth Edition By William Stallings Files Data collections created by users The File System is one of the most important parts

More information

Chapter 12: Mass-Storage Systems

Chapter 12: Mass-Storage Systems Chapter 12: Mass-Storage Systems Chapter 12: Mass-Storage Systems Overview of Mass Storage Structure Disk Structure Disk Attachment Disk Scheduling Disk Management Swap-Space Management RAID Structure

More information

CS2Bh: Current Technologies. Introduction to XML and Relational Databases. Introduction to Databases. Why databases? Why not use XML?

CS2Bh: Current Technologies. Introduction to XML and Relational Databases. Introduction to Databases. Why databases? Why not use XML? CS2Bh: Current Technologies Introduction to XML and Relational Databases Spring 2005 Introduction to Databases CS2 Spring 2005 (LN5) 1 Why databases? Why not use XML? What is missing from XML: Consistency

More information

Lecture 36: Chapter 6

Lecture 36: Chapter 6 Lecture 36: Chapter 6 Today s topic RAID 1 RAID Redundant Array of Inexpensive (Independent) Disks Use multiple smaller disks (c.f. one large disk) Parallelism improves performance Plus extra disk(s) for

More information

Platter. Track. Index Mark. Disk Storage. PHY 406F - Microprocessor Interfacing Techniques

Platter. Track. Index Mark. Disk Storage. PHY 406F - Microprocessor Interfacing Techniques Platter PHY 406F - icroprocessor Interfacing Techniques Disk Storage The major "permanent" storage medium for computers is, at present, generally magnetic media in the form of either magnetic tape or disks.

More information

Filing Systems. Filing Systems

Filing Systems. Filing Systems Filing Systems At the outset we identified long-term storage as desirable characteristic of an OS. EG: On-line storage for an MIS. Convenience of not having to re-write programs. Sharing of data in an

More information

Main Memory. Memory. Address binding. Memory spaces. All processes need main memory.

Main Memory. Memory. Address binding. Memory spaces. All processes need main memory. Memory If we define memory as a place where data is stored there are many levels of memory: Processor registers Primary (or main) memory RAM Secondary memory slower and more permanent disks Tertiary memory

More information

COMPUTER HARDWARE. Input- Output and Communication Memory Systems

COMPUTER HARDWARE. Input- Output and Communication Memory Systems COMPUTER HARDWARE Input- Output and Communication Memory Systems Computer I/O I/O devices commonly found in Computer systems Keyboards Displays Printers Magnetic Drives Compact disk read only memory (CD-ROM)

More information

William Stallings Computer Organization and Architecture 8 th Edition. External Memory

William Stallings Computer Organization and Architecture 8 th Edition. External Memory William Stallings Computer Organization and Architecture 8 th Edition Chapter 6 External Memory Types of External Memory Magnetic Disk RAID Removable Optical CD-ROM CD-Recordable (CD-R) CD-R/W DVD Magnetic

More information

Computer Systems Structure Main Memory Organization

Computer Systems Structure Main Memory Organization Computer Systems Structure Main Memory Organization Peripherals Computer Central Processing Unit Main Memory Computer Systems Interconnection Communication lines Input Output Ward 1 Ward 2 Storage/Memory

More information

Definition of RAID Levels

Definition of RAID Levels RAID The basic idea of RAID (Redundant Array of Independent Disks) is to combine multiple inexpensive disk drives into an array of disk drives to obtain performance, capacity and reliability that exceeds

More information

File-System Implementation

File-System Implementation File-System Implementation 11 CHAPTER In this chapter we discuss various methods for storing information on secondary storage. The basic issues are device directory, free space management, and space allocation

More information

Big Data Technology Map-Reduce Motivation: Indexing in Search Engines

Big Data Technology Map-Reduce Motivation: Indexing in Search Engines Big Data Technology Map-Reduce Motivation: Indexing in Search Engines Edward Bortnikov & Ronny Lempel Yahoo Labs, Haifa Indexing in Search Engines Information Retrieval s two main stages: Indexing process

More information

RAID HARDWARE. On board SATA RAID controller. RAID drive caddy (hot swappable) SATA RAID controller card. Anne Watson 1

RAID HARDWARE. On board SATA RAID controller. RAID drive caddy (hot swappable) SATA RAID controller card. Anne Watson 1 RAID HARDWARE On board SATA RAID controller SATA RAID controller card RAID drive caddy (hot swappable) Anne Watson 1 RAID The word redundant means an unnecessary repetition. The word array means a lineup.

More information

Review. Lecture 21: Reliable, High Performance Storage. Overview. Basic Disk & File System properties CSC 468 / CSC 2204 11/23/2006

Review. Lecture 21: Reliable, High Performance Storage. Overview. Basic Disk & File System properties CSC 468 / CSC 2204 11/23/2006 S 468 / S 2204 Review Lecture 2: Reliable, High Performance Storage S 469HF Fall 2006 ngela emke rown We ve looked at fault tolerance via server replication ontinue operating with up to f failures Recovery

More information

Case for storage. Outline. Magnetic disks. CS2410: Computer Architecture. Storage systems. Sangyeun Cho

Case for storage. Outline. Magnetic disks. CS2410: Computer Architecture. Storage systems. Sangyeun Cho Case for storage CS24: Computer Architecture Storage systems Sangyeun Cho Computer Science Department Shift in focus from computation to communication & storage of information Eg, Cray Research/Thinking

More information

Outline. Database Management and Tuning. Overview. Hardware Tuning. Johann Gamper. Unit 12

Outline. Database Management and Tuning. Overview. Hardware Tuning. Johann Gamper. Unit 12 Outline Database Management and Tuning Hardware Tuning Johann Gamper 1 Free University of Bozen-Bolzano Faculty of Computer Science IDSE Unit 12 2 3 Conclusion Acknowledgements: The slides are provided

More information

Hard Disk Drives and RAID

Hard Disk Drives and RAID Hard Disk Drives and RAID Janaka Harambearachchi (Engineer/Systems Development) INTERFACES FOR HDD A computer interfaces is what allows a computer to send and retrieve information for storage devices such

More information

Operating Systems. Virtual Memory

Operating Systems. Virtual Memory Operating Systems Virtual Memory Virtual Memory Topics. Memory Hierarchy. Why Virtual Memory. Virtual Memory Issues. Virtual Memory Solutions. Locality of Reference. Virtual Memory with Segmentation. Page

More information

Lecture 18: Reliable Storage

Lecture 18: Reliable Storage CS 422/522 Design & Implementation of Operating Systems Lecture 18: Reliable Storage Zhong Shao Dept. of Computer Science Yale University Acknowledgement: some slides are taken from previous versions of

More information

Network Attached Storage. Jinfeng Yang Oct/19/2015

Network Attached Storage. Jinfeng Yang Oct/19/2015 Network Attached Storage Jinfeng Yang Oct/19/2015 Outline Part A 1. What is the Network Attached Storage (NAS)? 2. What are the applications of NAS? 3. The benefits of NAS. 4. NAS s performance (Reliability

More information

Data storage Tree indexes

Data storage Tree indexes Data storage Tree indexes Rasmus Pagh February 7 lecture 1 Access paths For many database queries and updates, only a small fraction of the data needs to be accessed. Extreme examples are looking or updating

More information

With respect to the way of data access we can classify memories as:

With respect to the way of data access we can classify memories as: Memory Classification With respect to the way of data access we can classify memories as: - random access memories (RAM), - sequentially accessible memory (SAM), - direct access memory (DAM), - contents

More information

OS OBJECTIVE QUESTIONS

OS OBJECTIVE QUESTIONS OS OBJECTIVE QUESTIONS Which one of the following is Little s formula Where n is the average queue length, W is the time that a process waits 1)n=Lambda*W 2)n=Lambda/W 3)n=Lambda^W 4)n=Lambda*(W-n) Answer:1

More information

EMC CLARiiON RAID 6 Technology A Detailed Review

EMC CLARiiON RAID 6 Technology A Detailed Review A Detailed Review Abstract This white paper discusses the EMC CLARiiON RAID 6 implementation available in FLARE 26 and later, including an overview of RAID 6 and the CLARiiON-specific implementation, when

More information

PIONEER RESEARCH & DEVELOPMENT GROUP

PIONEER RESEARCH & DEVELOPMENT GROUP SURVEY ON RAID Aishwarya Airen 1, Aarsh Pandit 2, Anshul Sogani 3 1,2,3 A.I.T.R, Indore. Abstract RAID stands for Redundant Array of Independent Disk that is a concept which provides an efficient way for

More information

technology brief RAID Levels March 1997 Introduction Characteristics of RAID Levels

technology brief RAID Levels March 1997 Introduction Characteristics of RAID Levels technology brief RAID Levels March 1997 Introduction RAID is an acronym for Redundant Array of Independent Disks (originally Redundant Array of Inexpensive Disks) coined in a 1987 University of California

More information

Unit 4.3 - Storage Structures 1. Storage Structures. Unit 4.3

Unit 4.3 - Storage Structures 1. Storage Structures. Unit 4.3 Storage Structures Unit 4.3 Unit 4.3 - Storage Structures 1 The Physical Store Storage Capacity Medium Transfer Rate Seek Time Main Memory 800 MB/s 500 MB Instant Hard Drive 10 MB/s 120 GB 10 ms CD-ROM

More information