Skimpy-Stash. RAM Space Skimpy Key-Value Store on Flash-based Storage YASHWANTH BODDU FQ9316

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "Skimpy-Stash. RAM Space Skimpy Key-Value Store on Flash-based Storage YASHWANTH BODDU FQ9316"

Transcription

1 Skimpy-Stash RAM Space Skimpy Key-Value Store on Flash-based Storage YASHWANTH BODDU FQ9316

2 Outline Introduction Flash Memory Overview KV Store Applications Skimpy-Stash Design Architectural Components Hash Table Directory Design

3 Introduction This is designed to achieve high throughput and low latency. The distinguishing feature of Skimpy-Stash is the design goal of extremely low RAM footprint at about 1 (± 0.5) byte per key-value pair, which is more aggressive than earlier designs. It uses a hash table directory in RAM to index key value pairs stored in log structured manner on flash.

4 Q. Our base design uses less than 1 byte in RAM per key-value pair and our enhanced design takes slightly more than 1 byte per key-value pair. In FAWN, even a pointer to a KV pair needs a 4-byte pointer. How can possibly Skimpy-Stash achieve such a low memory cost for metadata? Skimpy-Stash moves most of the pointers that locate each key value pair from RAM to the Flash itself. Resolving hash table collisions using linear chaining and saving the linked lists on flash itself with a pointer in each hash table bucket in RAM pointing to the beginning record of the chain on flash. Enhanced design uses two-choice based load balancing to reduce wide variations in bucket sizes and a bloom filter in each hash table directory slot in RAM for summarizing the records in the bucket.

5 Q. Skimpy-Stash uses a hash table directory in RAM to index key-value pairs stored in a log-structure on flash. Why are key-value pairs on the flash organized as a log? The key benefit of the log structured data organization on the Flash may be the high write throughput that can be obtained, since all the updates to the data and the metadata are written in a sequential order in the log.

6 Q. The average bucket size is the critical design parameter that serves as a powerful knob for making a continuum of tradeoffs between low RAM usage and low lookup latencies. Please explain this statement. If the bucket size is small, the linked list will be greater in size and therefore we have more lookup latency. And, if the bucket size is more, linked list will be shorter in size and we have less lookup latency but more the size of the bucket, more RAM space will be utilized which is again a backdrop.

7 Flash Memory Overview

8 KV Store Applications Online Multi Player gaming There is a need to maintain server-side state so as to track player actions on each client machine and update global game states to make them visible to other players as quickly as possible. There is also the requirement to store server-side game state (i) resume game from interrupted state if and when crashes occur (ii) offline analysis of game popularity, progression, and dynamics with the objective of improving the game, and (iii) verification of player actions for fairness when outcomes are associated with monetary rewards. Storage Deduplication

9 Skimpy-Stash Design Coping with Flash Constraints Random Writes Writes less than Flash page size

10 Design Goals Support low latency, high throughput operations Use flash aware data structures and algorithms Random writes and in place updates are expensive on flash memory, hence they must be avoided and sequential writes should be used to extent possible. Low RAM footprint per key independent of key-value

11 Architectural Components RAM write buffer RAM Hash Table directory Flash Store

12 Q. The client [write] call returns only after the write buffer is flushed to flash. Why cannot such a call be acknowledged earlier? It can t be acknowledged earlier because the memory is still volatile in RAM write buffer.

13 Q. Basic functions: Store, Lookup, Delete explain how these basic functions are executed? Store: A key insert (or, update) operation (set) writes the key-value pair into the RAM write buffer. When there are enough key-value pairs in RAM write buffer to fill a flash page (or, a configurable timeout interval since the client call has expired, say 1 msec), these entries are written to flash and inserted into the RAM HT directory and flash. Lookup: It firsts looks up the RAM write buffer, upon a miss there it lookups the Hash Table directory in RAM and searches the chained key value pair records on flash in the respective bucket. Delete: A delete operation on a key is supported through insertion of null value for that key. Eventually the null entry and earlier inserted values of the key on flash will be garbage collected.

14 Q. The chain of records on flash pointed to by each slot comprises the bucket of records corresponding to this slot in the HT directory. Please use the figure to describe Skimpy-Stash s data structure. Also explain how lookup, insert, and delete operations are executed.

15 Hash Table Directory Design Base Design: Resolving hash table collisions using linear chaining, where multiple keys that resolve (collide) to the same hash table bucket are chained in a linked list, and Storing the linked lists on flash itself with a pointer in each hash table bucket in RAM pointing to the beginning record of the chain on flash. Each key-value pair record on flash contains, in addition to the key and value fields, a pointer to the next record (in the order in its respective chain) on flash.

16 Enhanced Design Load balancing across Buckets Bloom Filter per Bucket

17 Q. Because we store the chain of key-value pairs in each bucket on flash, we incur multiple flash reads upon lookup of a key in the store. Please explain how this issue can be alleviated. This can be done by periodically compacting the chain on flash in a bucket by placing the valid keys in the chain contiguously on one or more flash pages that are appended to the tail of log.

18 Q...two-choice based load balancing strategy is used to reduce variations in the number of keys assigned to each bucket. Explain how this is achieved. This strategy is used to reduce variations in the number of keys assigned to each bucket. With a load balanced design for HT directory, each key would be hashed to 2 candidate HT directory buckets using 2 hash functions h1 and h2, and actually inserted into the one that has currently fewer elements.

19 Q. when the last record in a bucket chain is encountered in the log during garbage collection, all valid records in that chain are compacted and relocated to the tail of the log.. Please explain how garbage is collected. When a certain configurable fraction of garbage accumulates in the log (in terms of space occupied), the pages on flash from the head of the log are recycled- valid entries from the head of the log are written back to end of the log while invalid entries can be skipped. This effectively leads to the design decision of garbage collecting entire bucket chains on flash at a time.

20 Thank You

SkimpyStash: RAM Space Skimpy Key-Value Store on Flash-based Storage

SkimpyStash: RAM Space Skimpy Key-Value Store on Flash-based Storage SkimpyStash: RAM Space Skimpy Key-Value Store on Flash-based Storage Biplob Debnath,1 Sudipta Sengupta Jin Li Microsoft Research, Redmond, WA, USA EMC Corporation, Santa Clara, CA, USA ABSTRACT We present

More information

Flash Memory Aware Software Architectures and Applications

Flash Memory Aware Software Architectures and Applications Flash Memory Aware Software Architectures and Applications Sudipta Sengupta and Jin Li Microsoft Research, Redmond, WA, USA Contains work that is joint with Biplob Debnath (Univ. of Minnesota) Flash Memory

More information

Speeding Up Cloud/Server Applications Using Flash Memory

Speeding Up Cloud/Server Applications Using Flash Memory Speeding Up Cloud/Server Applications Using Flash Memory Sudipta Sengupta Microsoft Research, Redmond, WA, USA Contains work that is joint with B. Debnath (Univ. of Minnesota) and J. Li (Microsoft Research,

More information

ChunkStash: Speeding up Inline Storage Deduplication using Flash Memory

ChunkStash: Speeding up Inline Storage Deduplication using Flash Memory ChunkStash: Speeding up Inline Storage Deduplication using Flash Memory Biplob Debnath Sudipta Sengupta Jin Li Microsoft Research, Redmond, WA, USA University of Minnesota, Twin Cities, USA Abstract Storage

More information

SMALL INDEX LARGE INDEX (SILT)

SMALL INDEX LARGE INDEX (SILT) Wayne State University ECE 7650: Scalable and Secure Internet Services and Architecture SMALL INDEX LARGE INDEX (SILT) A Memory Efficient High Performance Key Value Store QA REPORT Instructor: Dr. Song

More information

FAWN - a Fast Array of Wimpy Nodes

FAWN - a Fast Array of Wimpy Nodes University of Warsaw January 12, 2011 Outline Introduction 1 Introduction 2 3 4 5 Key issues Introduction Growing CPU vs. I/O gap Contemporary systems must serve millions of users Electricity consumed

More information

Benchmarking Cassandra on Violin

Benchmarking Cassandra on Violin Technical White Paper Report Technical Report Benchmarking Cassandra on Violin Accelerating Cassandra Performance and Reducing Read Latency With Violin Memory Flash-based Storage Arrays Version 1.0 Abstract

More information

COS 318: Operating Systems. Snapshot and NFS

COS 318: Operating Systems. Snapshot and NFS COS 318: Operating Systems Snapshot and NFS Andy Bavier Computer Science Department Princeton University http://www.cs.princeton.edu/courses/archive/fall10/cos318/ Topics Revisit Transactions and Logging

More information

A Deduplication File System & Course Review

A Deduplication File System & Course Review A Deduplication File System & Course Review Kai Li 12/13/12 Topics A Deduplication File System Review 12/13/12 2 Traditional Data Center Storage Hierarchy Clients Network Server SAN Storage Remote mirror

More information

Operating Systems CSE 410, Spring 2004. File Management. Stephen Wagner Michigan State University

Operating Systems CSE 410, Spring 2004. File Management. Stephen Wagner Michigan State University Operating Systems CSE 410, Spring 2004 File Management Stephen Wagner Michigan State University File Management File management system has traditionally been considered part of the operating system. Applications

More information

Benchmarking Hadoop & HBase on Violin

Benchmarking Hadoop & HBase on Violin Technical White Paper Report Technical Report Benchmarking Hadoop & HBase on Violin Harnessing Big Data Analytics at the Speed of Memory Version 1.0 Abstract The purpose of benchmarking is to show advantages

More information

A Data De-duplication Access Framework for Solid State Drives

A Data De-duplication Access Framework for Solid State Drives JOURNAL OF INFORMATION SCIENCE AND ENGINEERING 28, 941-954 (2012) A Data De-duplication Access Framework for Solid State Drives Department of Electronic Engineering National Taiwan University of Science

More information

A Hybrid Chaining Model with AVL and Binary Search Tree to Enhance Search Speed in Hashing

A Hybrid Chaining Model with AVL and Binary Search Tree to Enhance Search Speed in Hashing , pp.185-194 http://dx.doi.org/10.14257/ijhit.2015.8.3.18 A Hybrid Chaining Model with AVL and Binary Search Tree to Enhance Search Speed in Hashing 1 Akshay Saxena, 2 Harsh Anand, 3 Tribikram Pradhan

More information

Last Class: Memory Management. Recap: Paging

Last Class: Memory Management. Recap: Paging Last Class: Memory Management Static & Dynamic Relocation Fragmentation Paging Lecture 12, page 1 Recap: Paging Processes typically do not use their entire space in memory all the time. Paging 1. divides

More information

Copyright 2007 Ramez Elmasri and Shamkant B. Navathe. Slide 13-1

Copyright 2007 Ramez Elmasri and Shamkant B. Navathe. Slide 13-1 Slide 13-1 Chapter 13 Disk Storage, Basic File Structures, and Hashing Chapter Outline Disk Storage Devices Files of Records Operations on Files Unordered Files Ordered Files Hashed Files Dynamic and Extendible

More information

Raima Database Manager Version 14.0 In-memory Database Engine

Raima Database Manager Version 14.0 In-memory Database Engine + Raima Database Manager Version 14.0 In-memory Database Engine By Jeffrey R. Parsons, Senior Engineer January 2016 Abstract Raima Database Manager (RDM) v14.0 contains an all new data storage engine optimized

More information

Cuckoo Filter: Practically Better Than Bloom

Cuckoo Filter: Practically Better Than Bloom Cuckoo Filter: Practically Better Than Bloom Bin Fan, David G. Andersen, Michael Kaminsky, Michael D. Mitzenmacher Carnegie Mellon University, Intel Labs, Harvard University {binfan,dga}@cs.cmu.edu, michael.e.kaminsky@intel.com,

More information

Chapter 13 Disk Storage, Basic File Structures, and Hashing.

Chapter 13 Disk Storage, Basic File Structures, and Hashing. Chapter 13 Disk Storage, Basic File Structures, and Hashing. Copyright 2004 Pearson Education, Inc. Chapter Outline Disk Storage Devices Files of Records Operations on Files Unordered Files Ordered Files

More information

The Bw-Tree Key-Value Store and Its Applications to Server/Cloud Data Management in Production

The Bw-Tree Key-Value Store and Its Applications to Server/Cloud Data Management in Production The Bw-Tree Key-Value Store and Its Applications to Server/Cloud Data Management in Production Sudipta Sengupta Joint work with Justin Levandoski and David Lomet (Microsoft Research) And Microsoft Product

More information

The Google File System (GFS)

The Google File System (GFS) The Google File System (GFS) Google File System Example of clustered file system Basis of Hadoop s and Bigtable s underlying file system Many other implementations Design constraints Motivating application:

More information

Storage and File Structure

Storage and File Structure Storage and File Structure Chapter 10: Storage and File Structure Overview of Physical Storage Media Magnetic Disks RAID Tertiary Storage Storage Access File Organization Organization of Records in Files

More information

File System Management

File System Management Lecture 7: Storage Management File System Management Contents Non volatile memory Tape, HDD, SSD Files & File System Interface Directories & their Organization File System Implementation Disk Space Allocation

More information

Double-Take Pagefile Configuration

Double-Take Pagefile Configuration Double-Take Pagefile Configuration Double-Take Pagefile Configuration published August 2002 NSI and Double-Take are registered trademarks of Network Specialists, Inc. All other products are trademarks

More information

Hypertable Architecture Overview

Hypertable Architecture Overview WHITE PAPER - MARCH 2012 Hypertable Architecture Overview Hypertable is an open source, scalable NoSQL database modeled after Bigtable, Google s proprietary scalable database. It is written in C++ for

More information

ICS Principles of Operating Systems

ICS Principles of Operating Systems ICS 143 - Principles of Operating Systems Lectures 17-20 - FileSystem Interface and Implementation Prof. Ardalan Amiri Sani Prof. Nalini Venkatasubramanian ardalan@ics.uci.edu nalini@ics.uci.edu Outline

More information

WiscKey: Separating Keys from Values in SSD-Conscious Storage

WiscKey: Separating Keys from Values in SSD-Conscious Storage WiscKey: Separating Keys from Values in SSD-Conscious Storage Lanyue Lu, Thanumalayan Pillai, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau University of Wisconsin-Madison Key-Value Stores Key-Value

More information

File-System Implementation

File-System Implementation File-System Implementation 11 CHAPTER In this chapter we discuss various methods for storing information on secondary storage. The basic issues are device directory, free space management, and space allocation

More information

Chapter 13. Disk Storage, Basic File Structures, and Hashing

Chapter 13. Disk Storage, Basic File Structures, and Hashing Chapter 13 Disk Storage, Basic File Structures, and Hashing Chapter Outline Disk Storage Devices Files of Records Operations on Files Unordered Files Ordered Files Hashed Files Dynamic and Extendible Hashing

More information

the Nilfs version 1: overview

the Nilfs version 1: overview the Nilfs version 1: overview Nilfs team NTT Cyber Space Laboratories NTT Corporation http://www.osrg.net/nilfs/ nilfs@osrg.net 1 Introduction To enhance reliability of the Linux file system, we adopted

More information

Chapter 11: File System Implementation. Operating System Concepts with Java 8 th Edition

Chapter 11: File System Implementation. Operating System Concepts with Java 8 th Edition Chapter 11: File System Implementation 11.1 Silberschatz, Galvin and Gagne 2009 Chapter 11: File System Implementation File-System Structure File-System Implementation Directory Implementation Allocation

More information

DATABASE DESIGN - 1DL400

DATABASE DESIGN - 1DL400 DATABASE DESIGN - 1DL400 Spring 2015 A course on modern database systems!! http://www.it.uu.se/research/group/udbl/kurser/dbii_vt15/ Kjell Orsborn! Uppsala Database Laboratory! Department of Information

More information

Chapter 11: File System Implementation. Chapter 11: File System Implementation. Objectives. File-System Structure

Chapter 11: File System Implementation. Chapter 11: File System Implementation. Objectives. File-System Structure Chapter 11: File System Implementation Chapter 11: File System Implementation File-System Structure File-System Implementation Directory Implementation Allocation Methods Free-Space Management Efficiency

More information

6. Storage and File Structures

6. Storage and File Structures ECS-165A WQ 11 110 6. Storage and File Structures Goals Understand the basic concepts underlying different storage media, buffer management, files structures, and organization of records in files. Contents

More information

Improve Business Productivity and User Experience with a SanDisk Powered SQL Server 2014 In-Memory OLTP Database

Improve Business Productivity and User Experience with a SanDisk Powered SQL Server 2014 In-Memory OLTP Database WHITE PAPER Improve Business Productivity and User Experience with a SanDisk Powered SQL Server 2014 In-Memory OLTP Database 951 SanDisk Drive, Milpitas, CA 95035 www.sandisk.com Table of Contents Executive

More information

Storage in Database Systems. CMPSCI 445 Fall 2010

Storage in Database Systems. CMPSCI 445 Fall 2010 Storage in Database Systems CMPSCI 445 Fall 2010 1 Storage Topics Architecture and Overview Disks Buffer management Files of records 2 DBMS Architecture Query Parser Query Rewriter Query Optimizer Query

More information

Chapter 13 File and Database Systems

Chapter 13 File and Database Systems Chapter 13 File and Database Systems Outline 13.1 Introduction 13.2 Data Hierarchy 13.3 Files 13.4 File Systems 13.4.1 Directories 13.4. Metadata 13.4. Mounting 13.5 File Organization 13.6 File Allocation

More information

Chapter 13 File and Database Systems

Chapter 13 File and Database Systems Chapter 13 File and Database Systems Outline 13.1 Introduction 13.2 Data Hierarchy 13.3 Files 13.4 File Systems 13.4.1 Directories 13.4. Metadata 13.4. Mounting 13.5 File Organization 13.6 File Allocation

More information

TBF: A Memory-Efficient Replacement Policy for Flash-based Caches

TBF: A Memory-Efficient Replacement Policy for Flash-based Caches : A Memory-Efficient Replacement Policy for Flash-based Caches Cristian Ungureanu, Biplob Debnath, Stephen Rago, and Akshat Aranya NEC Laboratories America {cristian,biplob,sar,aranya}@nec-labs.com Abstract

More information

ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective

ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective Part II: Data Center Software Architecture: Topic 1: Distributed File Systems Finding a needle in Haystack: Facebook

More information

Physical Data Organization

Physical Data Organization Physical Data Organization Database design using logical model of the database - appropriate level for users to focus on - user independence from implementation details Performance - other major factor

More information

FAST 11. Yongseok Oh <ysoh@uos.ac.kr> University of Seoul. Mobile Embedded System Laboratory

FAST 11. Yongseok Oh <ysoh@uos.ac.kr> University of Seoul. Mobile Embedded System Laboratory CAFTL: A Content-Aware Flash Translation Layer Enhancing the Lifespan of flash Memory based Solid State Drives FAST 11 Yongseok Oh University of Seoul Mobile Embedded System Laboratory

More information

SILT: A Memory-Efficient, High-Performance Key-Value Store

SILT: A Memory-Efficient, High-Performance Key-Value Store SILT: A Memory-Efficient, High-Performance Key-Value Store Hyeontaek Lim Carnegie Mellon University David G. Andersen Carnegie Mellon University Bin Fan Carnegie Mellon University Michael Kaminsky Intel

More information

WITH A FUSION POWERED SQL SERVER 2014 IN-MEMORY OLTP DATABASE

WITH A FUSION POWERED SQL SERVER 2014 IN-MEMORY OLTP DATABASE WITH A FUSION POWERED SQL SERVER 2014 IN-MEMORY OLTP DATABASE 1 W W W. F U S I ON I O.COM Table of Contents Table of Contents... 2 Executive Summary... 3 Introduction: In-Memory Meets iomemory... 4 What

More information

File Management. Chapter 12

File Management. Chapter 12 Chapter 12 File Management File is the basic element of most of the applications, since the input to an application, as well as its output, is usually a file. They also typically outlive the execution

More information

SILT: A Memory-Efficient, High-Performance Key-Value Store

SILT: A Memory-Efficient, High-Performance Key-Value Store SILT: A Memory-Efficient, High-Performance Key-Value Store Hyeontaek Lim, Bin Fan, David G. Andersen, Michael Kaminsky Carnegie Mellon University, Intel Labs ABSTRACT SILT (Small Index Large Table) is

More information

Outline motivation hash functions collision handling. Courtesy to Goodrich, Tamassia and Olga Veksler 1

Outline motivation hash functions collision handling. Courtesy to Goodrich, Tamassia and Olga Veksler 1 Hash Table Outline motivation hash functions collision handling Courtesy to Goodrich, Tamassia and Olga Veksler Instructor: Yuzhen Xie 1 Log File vs Search Table Method Log File Search Table size, isempty

More information

COS 318: Operating Systems. File Layout and Directories. Vivek Pai Computer Science Department Princeton University

COS 318: Operating Systems. File Layout and Directories. Vivek Pai Computer Science Department Princeton University COS 318: Operating Systems File Layout and Directories Vivek Pai Computer Science Department Princeton University http://www.cs.princeton.edu/courses/archive/fall11/cos318/ Topics u File system structure

More information

ACM, 2012. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution.

ACM, 2012. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. ACM, 2012. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in SIGMOD Record, Volume

More information

Part III Storage Management. Chapter 11: File System Implementation

Part III Storage Management. Chapter 11: File System Implementation Part III Storage Management Chapter 11: File System Implementation 1 Layered File System 2 Overview: 1/4 A file system has on-disk and in-memory information. A disk may contain the following for implementing

More information

A SCALABLE DEDUPLICATION AND GARBAGE COLLECTION ENGINE FOR INCREMENTAL BACKUP

A SCALABLE DEDUPLICATION AND GARBAGE COLLECTION ENGINE FOR INCREMENTAL BACKUP A SCALABLE DEDUPLICATION AND GARBAGE COLLECTION ENGINE FOR INCREMENTAL BACKUP Dilip N Simha (Stony Brook University, NY & ITRI, Taiwan) Maohua Lu (IBM Almaden Research Labs, CA) Tzi-cker Chiueh (Stony

More information

A client side persistent block cache for the data center. Vault Boston 2015 - Luis Pabón - Red Hat

A client side persistent block cache for the data center. Vault Boston 2015 - Luis Pabón - Red Hat PBLCACHE A client side persistent block cache for the data center Vault Boston 2015 - Luis Pabón - Red Hat ABOUT ME LUIS PABÓN Principal Software Engineer, Red Hat Storage IRC, GitHub: lpabon QUESTIONS:

More information

Lecture 1: Data Storage & Index

Lecture 1: Data Storage & Index Lecture 1: Data Storage & Index R&G Chapter 8-11 Concurrency control Query Execution and Optimization Relational Operators File & Access Methods Buffer Management Disk Space Management Recovery Manager

More information

Comp215: Performance 2: Java Internals

Comp215: Performance 2: Java Internals Comp215: Performance 2: Java Internals Mack Joyner, Dan S. Wallach (Rice University) Copyright 2016, Mack Joyner, Dan S. Wallach. All rights reserved. Recall: Collision Resolution Chaining: 0 Store all

More information

Big Data Technology Map-Reduce Motivation: Indexing in Search Engines

Big Data Technology Map-Reduce Motivation: Indexing in Search Engines Big Data Technology Map-Reduce Motivation: Indexing in Search Engines Edward Bortnikov & Ronny Lempel Yahoo Labs, Haifa Indexing in Search Engines Information Retrieval s two main stages: Indexing process

More information

Finding a needle in Haystack: Facebook s photo storage IBM Haifa Research Storage Systems

Finding a needle in Haystack: Facebook s photo storage IBM Haifa Research Storage Systems Finding a needle in Haystack: Facebook s photo storage IBM Haifa Research Storage Systems 1 Some Numbers (2010) Over 260 Billion images (20 PB) 65 Billion X 4 different sizes for each image. 1 Billion

More information

Distributed File Systems

Distributed File Systems Distributed File Systems Paul Krzyzanowski Rutgers University October 28, 2012 1 Introduction The classic network file systems we examined, NFS, CIFS, AFS, Coda, were designed as client-server applications.

More information

COS 318: Operating Systems

COS 318: Operating Systems COS 318: Operating Systems File Performance and Reliability Andy Bavier Computer Science Department Princeton University http://www.cs.princeton.edu/courses/archive/fall10/cos318/ Topics File buffer cache

More information

Chapter 13. Chapter Outline. Disk Storage, Basic File Structures, and Hashing

Chapter 13. Chapter Outline. Disk Storage, Basic File Structures, and Hashing Chapter 13 Disk Storage, Basic File Structures, and Hashing Copyright 2007 Ramez Elmasri and Shamkant B. Navathe Chapter Outline Disk Storage Devices Files of Records Operations on Files Unordered Files

More information

Cache Mapping. COMP375 Computer Architecture and Organization

Cache Mapping. COMP375 Computer Architecture and Organization Cache Mapping COMP375 Computer Architecture and Organization The only problem in computer architecture that is really hard to overcome is not having enough address bits. Gordon Bell Exam on Wednesday The

More information

DNS LOOKUP SYSTEM DATA STRUCTURES AND ALGORITHMS PROJECT REPORT

DNS LOOKUP SYSTEM DATA STRUCTURES AND ALGORITHMS PROJECT REPORT DNS LOOKUP SYSTEM DATA STRUCTURES AND ALGORITHMS PROJECT REPORT By GROUP Avadhut Gurjar Mohsin Patel Shraddha Pandhe Page 1 Contents 1. Introduction... 3 2. DNS Recursive Query Mechanism:...5 2.1. Client

More information

A High-Throughput In-Memory Index, Durable on Flash-based SSD

A High-Throughput In-Memory Index, Durable on Flash-based SSD A High-Throughput In-Memory Index, Durable on Flash-based SSD Insights into the Winning Solution of the SIGMOD Programming Contest 2011 Thomas Kissinger, Benjamin Schlegel, Matthias Boehm, Dirk Habich,

More information

Couchbase Server Under the Hood

Couchbase Server Under the Hood Couchbase Server Under the Hood An Architectural Overview Couchbase Server is an open-source distributed NoSQL document-oriented database for interactive applications, uniquely suited for those needing

More information

EMC DATA DOMAIN SISL SCALING ARCHITECTURE

EMC DATA DOMAIN SISL SCALING ARCHITECTURE EMC DATA DOMAIN SISL SCALING ARCHITECTURE A Detailed Review ABSTRACT While tape has been the dominant storage medium for data protection for decades because of its low cost, it is steadily losing ground

More information

Oracle NoSQL Database and SanDisk Offer Cost-Effective Extreme Performance for Big Data

Oracle NoSQL Database and SanDisk Offer Cost-Effective Extreme Performance for Big Data WHITE PAPER Oracle NoSQL Database and SanDisk Offer Cost-Effective Extreme Performance for Big Data 951 SanDisk Drive, Milpitas, CA 95035 www.sandisk.com Table of Contents Abstract... 3 What Is Big Data?...

More information

CHAPTER 13: DISK STORAGE, BASIC FILE STRUCTURES, AND HASHING

CHAPTER 13: DISK STORAGE, BASIC FILE STRUCTURES, AND HASHING Chapter 13: Disk Storage, Basic File Structures, and Hashing 1 CHAPTER 13: DISK STORAGE, BASIC FILE STRUCTURES, AND HASHING Answers to Selected Exercises 13.23 Consider a disk with the following characteristics

More information

Heidi C. Ellis and Gerard C. Weatherby

Heidi C. Ellis and Gerard C. Weatherby Hashing Hashing: Applies transformation to keys to arrive at address of an element. * Performance is independent of table size. Perfect hash function: Each key is transformed into a unique storage location.

More information

CS 2112 Spring 2014. 0 Instructions. Assignment 3 Data Structures and Web Filtering. 0.1 Grading. 0.2 Partners. 0.3 Restrictions

CS 2112 Spring 2014. 0 Instructions. Assignment 3 Data Structures and Web Filtering. 0.1 Grading. 0.2 Partners. 0.3 Restrictions CS 2112 Spring 2014 Assignment 3 Data Structures and Web Filtering Due: March 4, 2014 11:59 PM Implementing spam blacklists and web filters requires matching candidate domain names and URLs very rapidly

More information

COS 318: Operating Systems. File Layout and Directories. Topics. File System Components. Steps to Open A File

COS 318: Operating Systems. File Layout and Directories. Topics. File System Components. Steps to Open A File Topics COS 318: Operating Systems File Layout and Directories File system structure Disk allocation and i-nodes Directory and link implementations Physical layout for performance 2 File System Components

More information

Data Structures for Big Data: Bloom Filter. Vinicius Vielmo Cogo Smalltalks, DI, FC/UL. October 16, 2014.

Data Structures for Big Data: Bloom Filter. Vinicius Vielmo Cogo Smalltalks, DI, FC/UL. October 16, 2014. Data Structures for Big Data: Bloom Filter Vinicius Vielmo Cogo Smalltalks, DI, FC/UL. October 16, 2014. is relative is not defined by a specific number of TB, PB, EB is when it becomes big for you is

More information

7. Memory Management

7. Memory Management Lecture Notes for CS347: Operating Systems Mythili Vutukuru, Department of Computer Science and Engineering, IIT Bombay 7. Memory Management 7.1 Basics of Memory Management What does main memory (RAM)

More information

Two Parts. Filesystem Interface. Filesystem design. Interface the user sees. Implementing the interface

Two Parts. Filesystem Interface. Filesystem design. Interface the user sees. Implementing the interface File Management Two Parts Filesystem Interface Interface the user sees Organization of the files as seen by the user Operations defined on files Properties that can be read/modified Filesystem design Implementing

More information

Quotient Filters: Approximate Membership Queries on the GPU

Quotient Filters: Approximate Membership Queries on the GPU Quotient Filters: Approximate Membership Queries on the GPU Afton Geil University of California, Davis GTC 2016 Outline What are approximate membership queries and how are they used? Background on quotient

More information

Universal hashing. In other words, the probability of a collision for two different keys x and y given a hash function randomly chosen from H is 1/m.

Universal hashing. In other words, the probability of a collision for two different keys x and y given a hash function randomly chosen from H is 1/m. Universal hashing No matter how we choose our hash function, it is always possible to devise a set of keys that will hash to the same slot, making the hash scheme perform poorly. To circumvent this, we

More information

Handling large amount of data efficiently

Handling large amount of data efficiently Handling large amount of data efficiently 1. Storage media and its constraints (magnetic disks, buffering) 2. Algorithms for large inputs (sorting) 3. Data structures (trees, hashes and bitmaps) Lecture

More information

NAND Flash Memories. Understanding NAND Flash Factory Pre-Programming. Schemes

NAND Flash Memories. Understanding NAND Flash Factory Pre-Programming. Schemes NAND Flash Memories Understanding NAND Flash Factory Pre-Programming Schemes Application Note February 2009 an_elnec_nand_schemes, version 1.00 Version 1.00/02.2009 Page 1 of 20 NAND flash technology enables

More information

BloomFlash: Bloom Filter on Flash-based Storage

BloomFlash: Bloom Filter on Flash-based Storage 211 31st International Conference on Distributed Computing Systems BloomFlash: Bloom Filter on Flash-based Storage Biplob Debnath,1, Sudipta Sengupta, Jin Li, David J. Lilja, David H.C. Du EMC Corporation,

More information

Record Storage and Primary File Organization

Record Storage and Primary File Organization Record Storage and Primary File Organization 1 C H A P T E R 4 Contents Introduction Secondary Storage Devices Buffering of Blocks Placing File Records on Disk Operations on Files Files of Unordered Records

More information

Distributed storage for structured data

Distributed storage for structured data Distributed storage for structured data Dennis Kafura CS5204 Operating Systems 1 Overview Goals scalability petabytes of data thousands of machines applicability to Google applications Google Analytics

More information

StreamStorage: High-throughput and Scalable Storage Technology for Streaming Data

StreamStorage: High-throughput and Scalable Storage Technology for Streaming Data : High-throughput and Scalable Storage Technology for Streaming Data Munenori Maeda Toshihiro Ozawa Real-time analytical processing (RTAP) of vast amounts of time-series data from sensors, server logs,

More information

Chapter 1 File Organization 1.0 OBJECTIVES 1.1 INTRODUCTION 1.2 STORAGE DEVICES CHARACTERISTICS

Chapter 1 File Organization 1.0 OBJECTIVES 1.1 INTRODUCTION 1.2 STORAGE DEVICES CHARACTERISTICS Chapter 1 File Organization 1.0 Objectives 1.1 Introduction 1.2 Storage Devices Characteristics 1.3 File Organization 1.3.1 Sequential Files 1.3.2 Indexing and Methods of Indexing 1.3.3 Hash Files 1.4

More information

Big Table A Distributed Storage System For Data

Big Table A Distributed Storage System For Data Big Table A Distributed Storage System For Data OSDI 2006 Fay Chang, Jeffrey Dean, Sanjay Ghemawat et.al. Presented by Rahul Malviya Why BigTable? Lots of (semi-)structured data at Google - - URLs: Contents,

More information

File Management Chapters 10, 11, 12

File Management Chapters 10, 11, 12 File Management Chapters 10, 11, 12 Requirements For long-term storage: possible to store large amount of info. info must survive termination of processes multiple processes must be able to access concurrently

More information

File Systems: Fundamentals

File Systems: Fundamentals Files What is a file? A named collection of related information recorded on secondary storage (e.g., disks) File Systems: Fundamentals File attributes Name, type, location, size, protection, creator, creation

More information

CS2210 Data Structures and Algorithms

CS2210 Data Structures and Algorithms CS2210 Data Structures and Algorithms Lecture 5: Hash Tables Instructor: Olga Veksler 0 1 2 3 025-612-0001 981-101-0002 4 451-229-0004 2004 Goodrich, Tamassia Outline Hash Tables Motivation Hash functions

More information

Carnegie Mellon Univ. Dept. of Computer Science Database Applications. Overview. Faloutsos CMU SCS

Carnegie Mellon Univ. Dept. of Computer Science Database Applications. Overview. Faloutsos CMU SCS Faloutsos 15-415 Carnegie Mellon Univ. Dept. of Computer Science 15-415 - Database Applications Lecture #8 (R&G ch9) Storing Data: Disks and Files Faloutsos 15-415 #1 Overview Memory hierarchy RAID (briefly)

More information

1. Comments on reviews a. Need to avoid just summarizing web page asks you for:

1. Comments on reviews a. Need to avoid just summarizing web page asks you for: 1. Comments on reviews a. Need to avoid just summarizing web page asks you for: i. A one or two sentence summary of the paper ii. A description of the problem they were trying to solve iii. A summary of

More information

Dictionary based on arrays. Hashing and Hash tables. Trade-offs. Hashing: Generalizing the index

Dictionary based on arrays. Hashing and Hash tables. Trade-offs. Hashing: Generalizing the index Dictionary based on arrays Hashing and Hash tables Suppose have a dictionary with small integer keys, in range -. Can use an array containing values, with key as index. private Object[] thedictionary =

More information

RAMCloud and the Low- Latency Datacenter. John Ousterhout Stanford University

RAMCloud and the Low- Latency Datacenter. John Ousterhout Stanford University RAMCloud and the Low- Latency Datacenter John Ousterhout Stanford University Most important driver for innovation in computer systems: Rise of the datacenter Phase 1: large scale Phase 2: low latency Introduction

More information

FPGA-based Multithreading for In-Memory Hash Joins

FPGA-based Multithreading for In-Memory Hash Joins FPGA-based Multithreading for In-Memory Hash Joins Robert J. Halstead, Ildar Absalyamov, Walid A. Najjar, Vassilis J. Tsotras University of California, Riverside Outline Background What are FPGAs Multithreaded

More information

University of Dublin Trinity College. File Organisation.

University of Dublin Trinity College. File Organisation. University of Dublin Trinity College File Organisation Owen.Conlan@cs.tcd.ie New records are inserted at the end of the file To search for a record linear search through the file records is necessary Typically

More information

Best Practices for Optimizing SQL Server Database Performance with the LSI WarpDrive Acceleration Card

Best Practices for Optimizing SQL Server Database Performance with the LSI WarpDrive Acceleration Card Best Practices for Optimizing SQL Server Database Performance with the LSI WarpDrive Acceleration Card Version 1.0 April 2011 DB15-000761-00 Revision History Version and Date Version 1.0, April 2011 Initial

More information

SDFS Overview. By Sam Silverberg

SDFS Overview. By Sam Silverberg SDFS Overview By Sam Silverberg Why did I do this? I had an Idea that I needed to see if it worked. Design Goals Create a dedup file system capable of effective inline deduplication for Virtual Machines

More information

Content: TDDB56 DALGOPT-D Algorithms and optimization. Lecture 4. Sets and Dictionaries Binary Search, Hashing, Skip Lists. ADT Set / Ordered Set

Content: TDDB56 DALGOPT-D Algorithms and optimization. Lecture 4. Sets and Dictionaries Binary Search, Hashing, Skip Lists. ADT Set / Ordered Set Content: TDDB56 DALGOPT-D Algorithms and optimization Lecture 4 Sets and Dictionaries Binary Search, Hashing, Skip Lists ADTs Set, Ordered Set, Dictionary/Map Implementation of Set and Dictionary using

More information

Snapshots in Hadoop Distributed File System

Snapshots in Hadoop Distributed File System Snapshots in Hadoop Distributed File System Sameer Agarwal UC Berkeley Dhruba Borthakur Facebook Inc. Ion Stoica UC Berkeley Abstract The ability to take snapshots is an essential functionality of any

More information

Storing Data: Disks and Files

Storing Data: Disks and Files Storing Data: Disks and Files [R&G] Chapter 9 CS 4320 1 Data on External Storage Disks: Can retrieve random page at fixed cost But reading several consecutive pages is much cheaper than reading them in

More information

J-Flow on J Series Services Routers and Branch SRX Series Services Gateways

J-Flow on J Series Services Routers and Branch SRX Series Services Gateways APPLICATION NOTE Juniper Flow Monitoring J-Flow on J Series Services Routers and Branch SRX Series Services Gateways Copyright 2011, Juniper Networks, Inc. 1 APPLICATION NOTE - Juniper Flow Monitoring

More information

Chapter 11: File System Implementation. Operating System Concepts 8 th Edition

Chapter 11: File System Implementation. Operating System Concepts 8 th Edition Chapter 11: File System Implementation Operating System Concepts 8 th Edition Silberschatz, Galvin and Gagne 2009 Chapter 11: File System Implementation File-System Structure File-System Implementation

More information

Operating Systems File Systems II

Operating Systems File Systems II CSCI-GA.2250-001 Operating Systems File Systems II Hubertus Franke frankeh@cs.nyu.edu Abstracted by OS as files A Conventional Hard Disk (Magnetic) Structure Hard Disk (Magnetic) Architecture Surface

More information

! Overview! Measures of Query Cost! Selection Operation! Sorting! Join Operation! Other Operations! Evaluation of Expressions

! Overview! Measures of Query Cost! Selection Operation! Sorting! Join Operation! Other Operations! Evaluation of Expressions Chapter 13: Query Processing Chapter 13: Query Processing! Overview! Measures of Query Cost! Selection Operation! Sorting! Join Operation! Other Operations! Evaluation of Expressions Basic Steps in Query

More information

Chapter 12 File Management

Chapter 12 File Management Operating Systems: Internals and Design Principles, 6/E William Stallings Chapter 12 File Management Patricia Roy Manatee Community College, Venice, FL 2008, Prentice Hall File Management File management

More information