ProTrack: A Simple Provenance-tracking Filesystem

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "ProTrack: A Simple Provenance-tracking Filesystem"

Transcription

1 ProTrack: A Simple Provenance-tracking Filesystem Somak Das Department of Electrical Engineering and Computer Science Massachusetts Institute of Technology Abstract Provenance describes a file s history. A user may want to discover the origin of a program on his system and study its source code or follow the propagation of mistakes in presentation slides and fix them. ProTrack is a simple yet effective provenance-tracking filesystem. Our approach divides a file into applicationspecific parts and stores provenance metadata in a hashtable. We believe querying file ancestry is ProTrack s most important feature, so we focus on optimizing traversals of persistent, on-disk data. The system design reduces forward and backward searches to fast lookups from within an inode. By storing parent/child links twice, we sacrifice some storage for speed, but being able to follow and manipulate dependency paths in either direction increases the responsiveness of our system. I. DESI GN A. File representation The main components of ProTrack on disk are the storage of versions, parts, provenance metadata, and parent directories for each file (see Figure 2). Versioning filesystem. To accurately track each file s provenance over time, the system needs an underlying filesystem that stores all versions of each file over time. Thus we build ProTrack on top of a versioning filesystem (VFS). We assume that the VFS creates separate inodes for different versions of the same file. File parts. ProTrack supports partitioning a file into a set of parts. Each part represents an applicationdefined unit of the file, such as a slide in a PowerPoint Presentation or an individual compressed file in a Zip archive. This enables fine-grained provenance tracking for the user: instead of hunting through an entire slide deck, he is able to quickly look up a particular slide s source. We implement parts by taking hints from the Unix FS. Each file s inode in Unix stores a set of device address pointing to data blocks on disk, where the last three point to indirect blocks for larger files. The ondisk inode in ProTrack stores an array of 32 numbered parts, which are each pointers to a Unix-style set of device addresses. The array index corresponds to the part number. Since some applications (like unmodified ones) will not define parts, Part 0 is reserved for file data that does not belong to any part. For files that need more parts, the system uses Unix s strategy of indirect layers the last (32 nd ) entry points to an indirect array of parts. This strategy does not adversely affect performance because, in practice, a device cache mechanism eliminates most of the indirect fetches. Provenance metadata. Each file stores its own provenance as lists of direct parents and children. In particular, the inode is augmented with a pointer to an int->provinfo hash-table on disk (with twice as many slots as parts and linear probing). This data structure, with amortized O(1) cost per lookup, maps the file s part numbers to their corresponding provenance information linked-lists of immediate ancestors and descendants, specified by (inode number, part number) pairs (see Figure 1). Fig. 1: Pseudocode for the ProvInfo data structure, which contains linked-lists of ProvNodes. struct ProvInfo { ProvNode *parents; ProvNode *children; } struct ProvNode { int inum, partnum; ProvInfo *next; } Parent directories. Provenance data stores inode and part numbers, but those are not user-friendly. While Unix supports converting file paths to i-numbers, the system should support reverse lookups too, from i- numbers to human-readable paths with filenames. Our simple and space-efficient solution is to only store i- numbers of the parent directories of the file s hard links inside the inode (note that parent directories refer to directories containing the file, not provenance) and recreate absolute paths with them. B. API implementation The implementation of ProTrack s system introduces new part and provenance operations, as well as functionality to existing file I/O operations. We designed the application programming interface to be minimal, so as to not introduce redundancy between calls. File I/O. ProTrack adds extra tracking functionality to the internals of standard, unchanged file I/O calls to support provenance. This allows an unmodified program, like a file copying utility, to update ancestor and descendant data as accurately as possible. When a process executes, the system updates provenance metadata by linking the files being read (parents) to the files being written to (children). The Unix kernel 1

2 today tracks a process s open files in an in-memory file descriptor table. ProTrack augments this existing table with read and write flags for each file (simple bits representing 0 for unflagged and 1 for flagged, see Figure 3). Since the process will probably call read() and write() many times, the system as a performance optimization only updates a written file s provenance once: when the process calls close() on it. The update consists of adding the files that were read to that file s list of immediate ancestors, using the link_prov() system call. The following file operations act like file I/O methods from Unix, with some additions: open(..) adds the file to the process s file descriptor table, with read and write unflagged. read(..) flags the file as read. write(..) flags the file as written. close(..) updates provenance of the file if it is written, making it dependent on all the files the process read (files in the table with read flagged) except those that are the file itself or are redundant with existing parents. The extended link() and unlink() system calls, which affect hard links, ensure that a file s list of parent directories is up-to-date. Since they use a lookup procedure that traverses over absolute paths, they can monitor the inode number of the directory containing that file. link(name, link_name) adds a new parent directory s i-number by examining the absolute path of link_name. unlink(name) removes an old parent directory s i- number by examining the absolute path of name. Part operations. ProTrack s part operations provide a simple interface for manipulating file parts, using Unix s existing file descriptors as a handle: create_part(fd) allocates a new part containing addresses pointing to data blocks. It returns the unique part number for the application to use for later operations. delete_part(fd, partnum)first updates provenance data of the file part s parents and children and then frees the part and its corresponding data blocks. The part number is now available for reuse. seek_part(fd, partnum) sets the file cursor to the beginning of the partnum th part. Since fd points the correct location, the application can use standard read() and write() system calls. Provenance operations. The supported provenance operations allow applications, along with system calls like close(), to update and query file history: Fig. 2: The contents of an inode in ProTrack. Note that it is augmented with file parts as well as information about provenance and parent directories. Fig. 3: A file descriptor table for a process copying src. file to dest.file. Flagged states are shaded green. As the blue arrow indicates, ProTrack s file I/O methods will eventually set the destination as the source s child and the source as the destination s parent. read_prov(fd, partnum, numlevels) finds the files on which the file part depends specifically, its ancestors up to the specified number of levels and returns a list of their i-numbers/parts. For example, if numlevels = 1, then calling read_prov() just returns the file s direct parents. Recall that Part 0 is special, so calling the function with partnum = 0 returns all the ancestors of that file as a whole. search_prov(fd, partnum, numlevels) similarly finds the files which depend on the file part descendants down to the specified number of levels and returns a list. link_prov(parentinum, parentpartnum, childinum, childpartnum) adds the child file part to the parent s list of immediate descendants and adds the parent file part to the child s list of immediate ancestors ensuring a complete, bidirectional mapping. unlink_prov(parentinum, parentpartnum, childinum, childpartnum) does the opposite of link_prov, removing. inode_number_to_paths(inum) converts an inode number to a list of human-readable paths that have hard links to the inode. It quickly resolves the i-number using the procedure in Figure 4. 2

3 Fig. 4: Pseudocode for the inode_numbers_to_path() system call inode_number_to_paths(inum) foreach parent directory stored in the inum inode Initialize current i-number = inum, current directory = parent directory while the current directory s parent is not itself (root) Look-up the current i-number in the current directory Prepend the filename found to the beginning of the path Update current i-number = current directory s i-number, current directory = current directory s parent return all unique paths found to inum Calls to read_prov() and search_prov() perform a backward and forward search, respectively, to traverse ancestors and descendants. Conceptually, the references to parents and children organize the VFS files into a graph. Thus the breadth-first search algorithm enables ProTrack to find all dependencies. With dynamic programming optimizations such as storing visited inodes to prevent unnecessary revisits, the search is time and space-efficient. File versions and garbage collection. For correct behavior, different versions of the same file over time have different provenance data. Since the VFS stores them as separate i-nodes, ProTrack stores the versions provenance in separate locations. When the VFS creates a new version of a file, it must call ProTrack to maintain the right relationships. ProTrack then copies the old file version s parents by value into the new version. An alternate implementation would just have a reference to the old version, but this approach is preferable. It does not rely on the old version staying alive in the FS, and it does not have long chains of files pointing to older versions of themselves. Our system does not assume a continuous VFS that keeps old versions of a file, because, in practice, many versions will be deleted to reclaim disk space ProTrack implements garbage collection based on this observation. When the VFS decides to delete an old file version (not necessarily when reference count decrements to zero), it must call ProTrack to immediately update the provenance of that file s dependencies. ProTrack then dynamically joins the file s ancestors to its descendants. (The same also applies to individual file parts.) This just-in-time garbage collection policy eliminates the need for scheduling a periodic process that does the same thus always keeping ProTrack consistent with the VFS. II. ANALYSI S A. Usage scenarios The following examples illustrate the behavior of ProTrack in four particular use cases. Copying files. ProTrack correctly updates provenance of a file even when the copy utility has not been modified to be provenance-aware. A program that copies file src to dest (from source to destination) continuously reads from src and writes to dest, which its file descriptor table knows via read/write flags. When it closes dest, ProTrack will correctly infer that dest depends on src and link them together as child and parent. We made the design decision to copy by reference (to src) instead of copy the provenance information by value. As a result, the user is able to track all the locations he copied src to. Compiling software. Similar to copy, ProTrack has full native support for the Make utility that builds executables and libraries from source code. Because Make reads source code and writes to intermediate files (like C object files), the intermediates become the source code s children. The resulting binaries, to which Make writes machine language, become children of the source and intermediate files because both were read during the compilation process. Copying PowerPoint slides. ProTrack introduces simple interfaces to read/write file parts and provenance data that provenance-aware applications can use. For example, PowerPoint can create a part for each slide, uniquely identified by a part number, and manipulate it with part operations. When the application detects that the user copied a slide from one presentation to another, it can: 1) Call link_prov() with the parent part and the child part. 2) Track the descendants of a particular slide by calling read_prov() with its PowerPoint file and part number, since ProTrack stores the provenance metadata of individual parts. 3) Use inode_number_to_paths() to produce human-readable paths of the descendant slides and display them to the user. The paths, along with part numbers, even allow PowerPoint to open parts in the background and show previews of those slides. Handling Zip archives. Using read_prov() and link_prov() on the files to be compressed, Zip can read, compress, decompress, and write their provenance data to preserve parent/child relationships between files in the Zip archive. We put the functionality of (a) filtering the lists of ancestors for files in the 3

4 archive and (b) marshalling provenance data in the application. First, Zip reads the parents of each file part, and if any of them will also go into the archive, then it stores the (file part, parent) relationship, replacing the i- numbers with unique archive IDs. Second, the un-zip application sees the provenance data as pairs of IDs, maps the archive IDs to the new i-numbers, and calls link_prov() to set up both parent and child relationships between each pair on disk. Due to file I/O system calls, the Zip archive itself is dependent on all the files it compressed, but the application ensures that the uncompressed files are not. Thus, this design propagates provenance by value. B. Performance analysis We analyze the system performance using the assumptions in Table I. Table I: Storage and timing assumptions Hard disk capacity 500GB Block size 0.5kB Number of files stored by the VFS 1,000,000 Parts per file 32 Parents per file part 8 Children per file part 8 Parent directories per file 4 CPU cycle 0.1ns Memory access 10ns Hard disk seek 10ms Hard disk latency per block 0.1ms Storage analysis. Overall, ProTrack uses less than 2% of the total disk space, even with storing the history of all file versions, bidirectional parent/child relationships, and assuming that every file uses 32 parts. The part data is the same as file data, so we do not consider it as ProTrack overhead. Note that the current system does not use any inmemory data structures except adding two bits to each row of file descriptor tables, which are negligible. Time analysis. Even though the provenance data is on disk, ProTrack stores the parent/child links twice, and these bidirectional links speed up the system call times. We now analyze two workloads, assuming hard disk drives to demonstrate the efficacy of our design. For a large FS, it should take a provenance-tracking system at most a few seconds to find all descendants of some file if there are only several (say, 8). In ProTrack, search_prov() performs a breadth-first search that easily surpasses this metric (see Table III). Table III: Timing the search_prov() system call Operation Worst-case time foreach descendant inode, including the start (8 + 1) Seek the inode, then seek its provenance metadata (2 10ms seek + Read its provenance metadata [0.1ms/0.5kB] (16B 32 (8 + 8) = 8kB full 8kB latency) hash-table) Total: 0.19s We can convert the return i-numbers to paths for an application within the reference time as well. If each path passes through 16 folders, and each Unix folder is ~2kB on average, then inode_number_to_paths (), without memoization, returns 8 paths in just 8 files 16 folders/absolute path (10ms + [0.1ms/0.5kB] 2kB) = 1.3s. ProTrack also sustains solid transfer rates in a continuous file copy workload. A usual 10kB file copy takes (2 10ms) + ([0.1ms/0.5kB] [2 10kB]) = 24ms disregarding the buffers in memory. ProTrack adds (2 10ms) + ([0.1ms/0.5kB] [8kB full hash-table + ~0kB new hash-table]) = 22ms of overhead, increasing the time to 46ms. The rate is thus 22 file copies per second. If the user was continuously deleting the original file as well, then the dominating step in the garbage collection would involve seeking the provenance of the old file s parents and new file (and joining them). Consequently the file transfer rate, assuming 8 parents, would be 5 file copies/second. Tradeoffs and scalability considerations. We faced two major design tradeoffs while building ProTrack. The first was interface simplicity vs. additional functionality. Our implementation of parts, using only an array of addresses, stays close to Unix and minimizes storage overhead. We did not want file parts to overly complicate the system s programming interface. Aligning with the end-to-end principle, extra features Table II: Disk space usage Item Size Number Cost File part pointers 4 bytes/address 32 1,000, MB Provenance metadata 16 bytes/i-number + part 32 (8 + 8) 1,000,000 8,192MB Hash-table overhead 32 bytes/file part 32 1,000,000 32MB Parent directories 8 bytes/i-number 4 1,000,000 4MB Total: 8,356MB 4

5 like naming parts with freeform strings are left out, although a particular application can easily define and change part names inside the file. The second was performance vs. on-disk storage. We sacrifice storage for duplicate parent/child links and a garbage collection policy that immediately joins the deleted file s parents to its children. This cost is offset by the speedups in bidirectional searches and full consistency between ProTrack and the VFS. Still, a main problem when accessing provenance is disk performance. We could complicate the system by storing fixed-size provenance in inode s xattrs and reduce the disk seeks. But seeks are becoming less of an issue as more users switch to solid-state drives, which effectively have zero seek times. Thus, we predict that the next step forward will be introducing in-memory data structures for provenance. IV. CONCLUSI ON S In summary, ProTrack offers simple yet fully functional support for provenance-tracking operations. The filesystem enables file history, versions, and parts for end users. It can support anticipated workflows such as building a PowerPoint Presentation or compiling source code, and meets all performance specifications. Future directions for this work include expanding provenance to multiple machines and improving performance with specialized provenance caches. Additional considerations that must be handled in the future are machine failures and security concerns, but otherwise, ProTrack is ready to be implemented. 5

Hypertable Architecture Overview

Hypertable Architecture Overview WHITE PAPER - MARCH 2012 Hypertable Architecture Overview Hypertable is an open source, scalable NoSQL database modeled after Bigtable, Google s proprietary scalable database. It is written in C++ for

More information

Two Parts. Filesystem Interface. Filesystem design. Interface the user sees. Implementing the interface

Two Parts. Filesystem Interface. Filesystem design. Interface the user sees. Implementing the interface File Management Two Parts Filesystem Interface Interface the user sees Organization of the files as seen by the user Operations defined on files Properties that can be read/modified Filesystem design Implementing

More information

The Linux Virtual Filesystem

The Linux Virtual Filesystem Lecture Overview Linux filesystem Linux virtual filesystem (VFS) overview Common file model Superblock, inode, file, dentry Object-oriented Ext2 filesystem Disk data structures Superblock, block group,

More information

Topics in Computer System Performance and Reliability: Storage Systems!

Topics in Computer System Performance and Reliability: Storage Systems! CSC 2233: Topics in Computer System Performance and Reliability: Storage Systems! Note: some of the slides in today s lecture are borrowed from a course taught by Greg Ganger and Garth Gibson at Carnegie

More information

File System Management

File System Management Lecture 7: Storage Management File System Management Contents Non volatile memory Tape, HDD, SSD Files & File System Interface Directories & their Organization File System Implementation Disk Space Allocation

More information

Chapter 11: File System Implementation. Operating System Concepts with Java 8 th Edition

Chapter 11: File System Implementation. Operating System Concepts with Java 8 th Edition Chapter 11: File System Implementation 11.1 Silberschatz, Galvin and Gagne 2009 Chapter 11: File System Implementation File-System Structure File-System Implementation Directory Implementation Allocation

More information

COS 318: Operating Systems

COS 318: Operating Systems COS 318: Operating Systems File Performance and Reliability Andy Bavier Computer Science Department Princeton University http://www.cs.princeton.edu/courses/archive/fall10/cos318/ Topics File buffer cache

More information

File Systems Management and Examples

File Systems Management and Examples File Systems Management and Examples Today! Efficiency, performance, recovery! Examples Next! Distributed systems Disk space management! Once decided to store a file as sequence of blocks What s the size

More information

Chapter 7. File system data structures. File system layout. Code: Block allocator

Chapter 7. File system data structures. File system layout. Code: Block allocator DRAFT as of November 19, 2010: Copyright 2009 Cox, Kaashoek, Morris Chapter 7 File system data structures The disk driver and buffer cache (Chapter 6) provide safe, synchronized access to disk blocks.

More information

COS 318: Operating Systems. File Layout and Directories. Topics. File System Components. Steps to Open A File

COS 318: Operating Systems. File Layout and Directories. Topics. File System Components. Steps to Open A File Topics COS 318: Operating Systems File Layout and Directories File system structure Disk allocation and i-nodes Directory and link implementations Physical layout for performance 2 File System Components

More information

SharePoint Server 2010 Capacity Management: Software Boundaries and Limits

SharePoint Server 2010 Capacity Management: Software Boundaries and Limits SharePoint Server 2010 Capacity Management: Software Boundaries and s This document is provided as-is. Information and views expressed in this document, including URL and other Internet Web site references,

More information

COSC 6374 Parallel Computation. Parallel I/O (I) I/O basics. Concept of a clusters

COSC 6374 Parallel Computation. Parallel I/O (I) I/O basics. Concept of a clusters COSC 6374 Parallel I/O (I) I/O basics Fall 2012 Concept of a clusters Processor 1 local disks Compute node message passing network administrative network Memory Processor 2 Network card 1 Network card

More information

tmpfs: A Virtual Memory File System

tmpfs: A Virtual Memory File System tmpfs: A Virtual Memory File System Peter Snyder Sun Microsystems Inc. 2550 Garcia Avenue Mountain View, CA 94043 ABSTRACT This paper describes tmpfs, a memory-based file system that uses resources and

More information

Network File System (NFS)

Network File System (NFS) Network File System (NFS) Brad Karp UCL Computer Science CS GZ03 / M030 10 th October 2011 NFS Is Relevant Original paper from 1985 Very successful, still widely used today Early result; much subsequent

More information

Cloud Computing at Google. Architecture

Cloud Computing at Google. Architecture Cloud Computing at Google Google File System Web Systems and Algorithms Google Chris Brooks Department of Computer Science University of San Francisco Google has developed a layered system to handle webscale

More information

Distributed File Systems

Distributed File Systems Distributed File Systems Paul Krzyzanowski Rutgers University October 28, 2012 1 Introduction The classic network file systems we examined, NFS, CIFS, AFS, Coda, were designed as client-server applications.

More information

TECHNICAL BRIEF. Primary Storage Compression with Storage Foundation 6.0

TECHNICAL BRIEF. Primary Storage Compression with Storage Foundation 6.0 TECHNICAL BRIEF Primary Storage Compression with Storage Foundation 6.0 Technical Brief Primary Storage Compression with Storage Foundation 6.0 Contents Introduction... 4 What is Compression?... 4 Differentiators...

More information

Chapter 13 File and Database Systems

Chapter 13 File and Database Systems Chapter 13 File and Database Systems Outline 13.1 Introduction 13.2 Data Hierarchy 13.3 Files 13.4 File Systems 13.4.1 Directories 13.4. Metadata 13.4. Mounting 13.5 File Organization 13.6 File Allocation

More information

Chapter 13 File and Database Systems

Chapter 13 File and Database Systems Chapter 13 File and Database Systems Outline 13.1 Introduction 13.2 Data Hierarchy 13.3 Files 13.4 File Systems 13.4.1 Directories 13.4. Metadata 13.4. Mounting 13.5 File Organization 13.6 File Allocation

More information

Chapter 6, The Operating System Machine Level

Chapter 6, The Operating System Machine Level Chapter 6, The Operating System Machine Level 6.1 Virtual Memory 6.2 Virtual I/O Instructions 6.3 Virtual Instructions For Parallel Processing 6.4 Example Operating Systems 6.5 Summary Virtual Memory General

More information

CS 377: Operating Systems. Outline. A review of what you ve learned, and how it applies to a real operating system. Lecture 25 - Linux Case Study

CS 377: Operating Systems. Outline. A review of what you ve learned, and how it applies to a real operating system. Lecture 25 - Linux Case Study CS 377: Operating Systems Lecture 25 - Linux Case Study Guest Lecturer: Tim Wood Outline Linux History Design Principles System Overview Process Scheduling Memory Management File Systems A review of what

More information

Windows NT File System. Outline. Hardware Basics. Ausgewählte Betriebssysteme Institut Betriebssysteme Fakultät Informatik

Windows NT File System. Outline. Hardware Basics. Ausgewählte Betriebssysteme Institut Betriebssysteme Fakultät Informatik Windows Ausgewählte Betriebssysteme Institut Betriebssysteme Fakultät Informatik Outline NTFS File System Formats File System Driver Architecture Advanced Features NTFS Driver On-Disk Structure (MFT,...)

More information

SAP HANA - Main Memory Technology: A Challenge for Development of Business Applications. Jürgen Primsch, SAP AG July 2011

SAP HANA - Main Memory Technology: A Challenge for Development of Business Applications. Jürgen Primsch, SAP AG July 2011 SAP HANA - Main Memory Technology: A Challenge for Development of Business Applications Jürgen Primsch, SAP AG July 2011 Why In-Memory? Information at the Speed of Thought Imagine access to business data,

More information

Outline. Windows NT File System. Hardware Basics. Win2K File System Formats. NTFS Cluster Sizes NTFS

Outline. Windows NT File System. Hardware Basics. Win2K File System Formats. NTFS Cluster Sizes NTFS Windows Ausgewählte Betriebssysteme Institut Betriebssysteme Fakultät Informatik 2 Hardware Basics Win2K File System Formats Sector: addressable block on storage medium usually 512 bytes (x86 disks) Cluster:

More information

Network Attached Storage. Jinfeng Yang Oct/19/2015

Network Attached Storage. Jinfeng Yang Oct/19/2015 Network Attached Storage Jinfeng Yang Oct/19/2015 Outline Part A 1. What is the Network Attached Storage (NAS)? 2. What are the applications of NAS? 3. The benefits of NAS. 4. NAS s performance (Reliability

More information

Lecture 10: Dynamic Memory Allocation 1: Into the jaws of malloc()

Lecture 10: Dynamic Memory Allocation 1: Into the jaws of malloc() CS61: Systems Programming and Machine Organization Harvard University, Fall 2009 Lecture 10: Dynamic Memory Allocation 1: Into the jaws of malloc() Prof. Matt Welsh October 6, 2009 Topics for today Dynamic

More information

Chapter 12 File Management

Chapter 12 File Management Operating Systems: Internals and Design Principles, 6/E William Stallings Chapter 12 File Management Dave Bremer Otago Polytechnic, N.Z. 2008, Prentice Hall Roadmap Overview File organisation and Access

More information

Chapter 12 File Management. Roadmap

Chapter 12 File Management. Roadmap Operating Systems: Internals and Design Principles, 6/E William Stallings Chapter 12 File Management Dave Bremer Otago Polytechnic, N.Z. 2008, Prentice Hall Overview Roadmap File organisation and Access

More information

PART IV Performance oriented design, Performance testing, Performance tuning & Performance solutions. Outline. Performance oriented design

PART IV Performance oriented design, Performance testing, Performance tuning & Performance solutions. Outline. Performance oriented design PART IV Performance oriented design, Performance testing, Performance tuning & Performance solutions Slide 1 Outline Principles for performance oriented design Performance testing Performance tuning General

More information

Cache Configuration Reference

Cache Configuration Reference Sitecore CMS 6.2 Cache Configuration Reference Rev: 2009-11-20 Sitecore CMS 6.2 Cache Configuration Reference Tips and Techniques for Administrators and Developers Table of Contents Chapter 1 Introduction...

More information

CSE 120 Principles of Operating Systems

CSE 120 Principles of Operating Systems CSE 120 Principles of Operating Systems Fall 2004 Lecture 13: FFS, LFS, RAID Geoffrey M. Voelker Overview We ve looked at disks and file systems generically Now we re going to look at some example file

More information

File System Implementation II

File System Implementation II Introduction to Operating Systems File System Implementation II Performance, Recovery, Network File System John Franco Electrical Engineering and Computing Systems University of Cincinnati Review Block

More information

Simple Solution for a Location Service. Naming vs. Locating Entities. Forwarding Pointers (2) Forwarding Pointers (1)

Simple Solution for a Location Service. Naming vs. Locating Entities. Forwarding Pointers (2) Forwarding Pointers (1) Naming vs. Locating Entities Till now: resources with fixed locations (hierarchical, caching,...) Problem: some entity may change its location frequently Simple solution: record aliases for the new address

More information

Unix Filesystem Organization

Unix Filesystem Organization Unix Filesystem Organization Old (Original) file system In the original Unix file system, Unix divided physical disks into logical disks called partitions. Each partition is a standalone file system. We

More information

Binary search tree with SIMD bandwidth optimization using SSE

Binary search tree with SIMD bandwidth optimization using SSE Binary search tree with SIMD bandwidth optimization using SSE Bowen Zhang, Xinwei Li 1.ABSTRACT In-memory tree structured index search is a fundamental database operation. Modern processors provide tremendous

More information

DEXT3: Block Level Inline Deduplication for EXT3 File System

DEXT3: Block Level Inline Deduplication for EXT3 File System DEXT3: Block Level Inline Deduplication for EXT3 File System Amar More M.A.E. Alandi, Pune, India ahmore@comp.maepune.ac.in Zishan Shaikh M.A.E. Alandi, Pune, India zishan366shaikh@gmail.com Vishal Salve

More information

COSC 6374 Parallel Computation. Parallel I/O (I) I/O basics. Concept of a clusters

COSC 6374 Parallel Computation. Parallel I/O (I) I/O basics. Concept of a clusters COSC 6374 Parallel Computation Parallel I/O (I) I/O basics Spring 2008 Concept of a clusters Processor 1 local disks Compute node message passing network administrative network Memory Processor 2 Network

More information

Chapter 11: File System Implementation. Operating System Concepts 8 th Edition

Chapter 11: File System Implementation. Operating System Concepts 8 th Edition Chapter 11: File System Implementation Operating System Concepts 8 th Edition Silberschatz, Galvin and Gagne 2009 Chapter 11: File System Implementation File-System Structure File-System Implementation

More information

Chapter 12 File Management

Chapter 12 File Management Operating Systems: Internals and Design Principles Chapter 12 File Management Eighth Edition By William Stallings Files Data collections created by users The File System is one of the most important parts

More information

Original-page small file oriented EXT3 file storage system

Original-page small file oriented EXT3 file storage system Original-page small file oriented EXT3 file storage system Zhang Weizhe, Hui He, Zhang Qizhen School of Computer Science and Technology, Harbin Institute of Technology, Harbin E-mail: wzzhang@hit.edu.cn

More information

Raima Database Manager Version 14.0 In-memory Database Engine

Raima Database Manager Version 14.0 In-memory Database Engine + Raima Database Manager Version 14.0 In-memory Database Engine By Jeffrey R. Parsons, Senior Engineer January 2016 Abstract Raima Database Manager (RDM) v14.0 contains an all new data storage engine optimized

More information

Workflow Templates Library

Workflow Templates Library Workflow s Library Table of Contents Intro... 2 Active Directory... 3 Application... 5 Cisco... 7 Database... 8 Excel Automation... 9 Files and Folders... 10 FTP Tasks... 13 Incident Management... 14 Security

More information

Lecture 5: GFS & HDFS! Claudia Hauff (Web Information Systems)! ti2736b-ewi@tudelft.nl

Lecture 5: GFS & HDFS! Claudia Hauff (Web Information Systems)! ti2736b-ewi@tudelft.nl Big Data Processing, 2014/15 Lecture 5: GFS & HDFS!! Claudia Hauff (Web Information Systems)! ti2736b-ewi@tudelft.nl 1 Course content Introduction Data streams 1 & 2 The MapReduce paradigm Looking behind

More information

CHAPTER 17: File Management

CHAPTER 17: File Management CHAPTER 17: File Management The Architecture of Computer Hardware, Systems Software & Networking: An Information Technology Approach 4th Edition, Irv Englander John Wiley and Sons 2010 PowerPoint slides

More information

Datacenter Operating Systems

Datacenter Operating Systems Datacenter Operating Systems CSE451 Simon Peter With thanks to Timothy Roscoe (ETH Zurich) Autumn 2015 This Lecture What s a datacenter Why datacenters Types of datacenters Hyperscale datacenters Major

More information

A Comparative Study on Vega-HTTP & Popular Open-source Web-servers

A Comparative Study on Vega-HTTP & Popular Open-source Web-servers A Comparative Study on Vega-HTTP & Popular Open-source Web-servers Happiest People. Happiest Customers Contents Abstract... 3 Introduction... 3 Performance Comparison... 4 Architecture... 5 Diagram...

More information

Operating Systems CSE 410, Spring 2004. File Management. Stephen Wagner Michigan State University

Operating Systems CSE 410, Spring 2004. File Management. Stephen Wagner Michigan State University Operating Systems CSE 410, Spring 2004 File Management Stephen Wagner Michigan State University File Management File management system has traditionally been considered part of the operating system. Applications

More information

Chapter 11: File-System Interface

Chapter 11: File-System Interface Chapter 11: File-System Interface Chapter Outline File Concept Access Methods Directory Structure File System Mounting File Sharing Protection 11.1 File Systems File System consists of A collection of

More information

Naming vs. Locating Entities

Naming vs. Locating Entities Naming vs. Locating Entities Till now: resources with fixed locations (hierarchical, caching,...) Problem: some entity may change its location frequently Simple solution: record aliases for the new address

More information

Review. Lecture 21: Reliable, High Performance Storage. Overview. Basic Disk & File System properties CSC 468 / CSC 2204 11/23/2006

Review. Lecture 21: Reliable, High Performance Storage. Overview. Basic Disk & File System properties CSC 468 / CSC 2204 11/23/2006 S 468 / S 2204 Review Lecture 2: Reliable, High Performance Storage S 469HF Fall 2006 ngela emke rown We ve looked at fault tolerance via server replication ontinue operating with up to f failures Recovery

More information

Survey of Filesystems for Embedded Linux. Presented by Gene Sally CELF

Survey of Filesystems for Embedded Linux. Presented by Gene Sally CELF Survey of Filesystems for Embedded Linux Presented by Gene Sally CELF Presentation Filesystems In Summary What is a filesystem Kernel and User space filesystems Picking a root filesystem Filesystem Round-up

More information

CS 153 Design of Operating Systems Spring 2015

CS 153 Design of Operating Systems Spring 2015 CS 153 Design of Operating Systems Spring 2015 Lecture 22: File system optimizations Physical Disk Structure Disk components Platters Surfaces Tracks Arm Track Sector Surface Sectors Cylinders Arm Heads

More information

Department of Electrical Engineering and Computer Science MASSACHUSETTS INSTITUTE OF TECHNOLOGY. 6.828 Operating System Engineering: Fall 2005

Department of Electrical Engineering and Computer Science MASSACHUSETTS INSTITUTE OF TECHNOLOGY. 6.828 Operating System Engineering: Fall 2005 Department of Electrical Engineering and Computer Science MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.828 Operating System Engineering: Fall 2005 Quiz II Solutions Average 84, median 83, standard deviation

More information

EMC VNXe File Deduplication and Compression

EMC VNXe File Deduplication and Compression White Paper EMC VNXe File Deduplication and Compression Overview Abstract This white paper describes EMC VNXe File Deduplication and Compression, a VNXe system feature that increases the efficiency with

More information

Virtual Machine Monitors. Dr. Marc E. Fiuczynski Research Scholar Princeton University

Virtual Machine Monitors. Dr. Marc E. Fiuczynski Research Scholar Princeton University Virtual Machine Monitors Dr. Marc E. Fiuczynski Research Scholar Princeton University Introduction Have been around since 1960 s on mainframes used for multitasking Good example VM/370 Have resurfaced

More information

We mean.network File System

We mean.network File System We mean.network File System Introduction: Remote File-systems When networking became widely available users wanting to share files had to log in across the net to a central machine This central machine

More information

Network File System (NFS) Pradipta De pradipta.de@sunykorea.ac.kr

Network File System (NFS) Pradipta De pradipta.de@sunykorea.ac.kr Network File System (NFS) Pradipta De pradipta.de@sunykorea.ac.kr Today s Topic Network File System Type of Distributed file system NFS protocol NFS cache consistency issue CSE506: Ext Filesystem 2 NFS

More information

File Systems for Flash Memories. Marcela Zuluaga Sebastian Isaza Dante Rodriguez

File Systems for Flash Memories. Marcela Zuluaga Sebastian Isaza Dante Rodriguez File Systems for Flash Memories Marcela Zuluaga Sebastian Isaza Dante Rodriguez Outline Introduction to Flash Memories Introduction to File Systems File Systems for Flash Memories YAFFS (Yet Another Flash

More information

09'Linux Plumbers Conference

09'Linux Plumbers Conference 09'Linux Plumbers Conference Data de duplication Mingming Cao IBM Linux Technology Center cmm@us.ibm.com 2009 09 25 Current storage challenges Our world is facing data explosion. Data is growing in a amazing

More information

File-System Implementation

File-System Implementation File-System Implementation 11 CHAPTER In this chapter we discuss various methods for storing information on secondary storage. The basic issues are device directory, free space management, and space allocation

More information

Prof. Dr. Ing. Axel Hunger Dipl.-Ing. Bogdan Marin. Operation Systems and Computer Networks Betriebssysteme und Computer Netzwerke

Prof. Dr. Ing. Axel Hunger Dipl.-Ing. Bogdan Marin. Operation Systems and Computer Networks Betriebssysteme und Computer Netzwerke Ex 2 File Systems A file is a logical collection of information and a file system is a collection of files, where the latter may also include a variety of other objects that share many of the properties

More information

Today s Papers. RAID Basics (Two optional papers) Array Reliability. EECS 262a Advanced Topics in Computer Systems Lecture 4

Today s Papers. RAID Basics (Two optional papers) Array Reliability. EECS 262a Advanced Topics in Computer Systems Lecture 4 EECS 262a Advanced Topics in Computer Systems Lecture 4 Filesystems (Con t) September 15 th, 2014 John Kubiatowicz Electrical Engineering and Computer Sciences University of California, Berkeley Today

More information

Outline: Operating Systems

Outline: Operating Systems Outline: Operating Systems What is an OS OS Functions Multitasking Virtual Memory File Systems Window systems PC Operating System Wars: Windows vs. Linux 1 Operating System provides a way to boot (start)

More information

Chapter 11 I/O Management and Disk Scheduling

Chapter 11 I/O Management and Disk Scheduling Operating Systems: Internals and Design Principles, 6/E William Stallings Chapter 11 I/O Management and Disk Scheduling Dave Bremer Otago Polytechnic, NZ 2008, Prentice Hall I/O Devices Roadmap Organization

More information

Introduction to Hadoop HDFS and Ecosystems. Slides credits: Cloudera Academic Partners Program & Prof. De Liu, MSBA 6330 Harvesting Big Data

Introduction to Hadoop HDFS and Ecosystems. Slides credits: Cloudera Academic Partners Program & Prof. De Liu, MSBA 6330 Harvesting Big Data Introduction to Hadoop HDFS and Ecosystems ANSHUL MITTAL Slides credits: Cloudera Academic Partners Program & Prof. De Liu, MSBA 6330 Harvesting Big Data Topics The goal of this presentation is to give

More information

The Classical Architecture. Storage 1 / 36

The Classical Architecture. Storage 1 / 36 1 / 36 The Problem Application Data? Filesystem Logical Drive Physical Drive 2 / 36 Requirements There are different classes of requirements: Data Independence application is shielded from physical storage

More information

Windows NT File System. Ausgewählte Betriebssysteme Institut Betriebssysteme Fakultät Informatik

Windows NT File System. Ausgewählte Betriebssysteme Institut Betriebssysteme Fakultät Informatik Windows Ausgewählte Betriebssysteme Institut Betriebssysteme Fakultät Informatik Outline NTFS File System Formats File System Driver Architecture Advanced Features NTFS Driver On-Disk Structure (MFT,...)

More information

FAST 11. Yongseok Oh University of Seoul. Mobile Embedded System Laboratory

FAST 11. Yongseok Oh <ysoh@uos.ac.kr> University of Seoul. Mobile Embedded System Laboratory CAFTL: A Content-Aware Flash Translation Layer Enhancing the Lifespan of flash Memory based Solid State Drives FAST 11 Yongseok Oh University of Seoul Mobile Embedded System Laboratory

More information

12 Abstract Data Types

12 Abstract Data Types 12 Abstract Data Types 12.1 Source: Foundations of Computer Science Cengage Learning Objectives After studying this chapter, the student should be able to: Define the concept of an abstract data type (ADT).

More information

Chapter 17: Distributed-File Systems. Operating System Concepts 8 th Edition,

Chapter 17: Distributed-File Systems. Operating System Concepts 8 th Edition, Chapter 17: Distributed-File Systems, Silberschatz, Galvin and Gagne 2009 Chapter 17 Distributed-File Systems Background Naming and Transparency Remote File Access Stateful versus Stateless Service File

More information

Operating Systems: Internals and Design Principles. Chapter 12 File Management Seventh Edition By William Stallings

Operating Systems: Internals and Design Principles. Chapter 12 File Management Seventh Edition By William Stallings Operating Systems: Internals and Design Principles Chapter 12 File Management Seventh Edition By William Stallings Operating Systems: Internals and Design Principles If there is one singular characteristic

More information

ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective

ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective Part II: Data Center Software Architecture: Topic 1: Distributed File Systems Finding a needle in Haystack: Facebook

More information

A Data De-duplication Access Framework for Solid State Drives

A Data De-duplication Access Framework for Solid State Drives JOURNAL OF INFORMATION SCIENCE AND ENGINEERING 28, 941-954 (2012) A Data De-duplication Access Framework for Solid State Drives Department of Electronic Engineering National Taiwan University of Science

More information

& Data Processing 2. Exercise 2: File Systems. Dipl.-Ing. Bogdan Marin. Universität Duisburg-Essen

& Data Processing 2. Exercise 2: File Systems. Dipl.-Ing. Bogdan Marin. Universität Duisburg-Essen Folie a: Name & Data Processing 2 2: File Systems Dipl.-Ing. Bogdan Marin Fakultät für Ingenieurwissenschaften Abteilung Elektro-und Informationstechnik -Technische Informatik- Objectives File System Concept

More information

1 File Management. 1.1 Naming. COMP 242 Class Notes Section 6: File Management

1 File Management. 1.1 Naming. COMP 242 Class Notes Section 6: File Management COMP 242 Class Notes Section 6: File Management 1 File Management We shall now examine how an operating system provides file management. We shall define a file to be a collection of permanent data with

More information

Secondary Storage. Any modern computer system will incorporate (at least) two levels of storage: magnetic disk/optical devices/tape systems

Secondary Storage. Any modern computer system will incorporate (at least) two levels of storage: magnetic disk/optical devices/tape systems 1 Any modern computer system will incorporate (at least) two levels of storage: primary storage: typical capacity cost per MB $3. typical access time burst transfer rate?? secondary storage: typical capacity

More information

COS 318: Operating Systems. Virtual Machine Monitors

COS 318: Operating Systems. Virtual Machine Monitors COS 318: Operating Systems Virtual Machine Monitors Kai Li and Andy Bavier Computer Science Department Princeton University http://www.cs.princeton.edu/courses/archive/fall13/cos318/ Introduction u Have

More information

Storage in Database Systems. CMPSCI 445 Fall 2010

Storage in Database Systems. CMPSCI 445 Fall 2010 Storage in Database Systems CMPSCI 445 Fall 2010 1 Storage Topics Architecture and Overview Disks Buffer management Files of records 2 DBMS Architecture Query Parser Query Rewriter Query Optimizer Query

More information

Chapter 11: File System Implementation. Chapter 11: File System Implementation. Objectives. File-System Structure

Chapter 11: File System Implementation. Chapter 11: File System Implementation. Objectives. File-System Structure Chapter 11: File System Implementation Chapter 11: File System Implementation File-System Structure File-System Implementation Directory Implementation Allocation Methods Free-Space Management Efficiency

More information

Big Data With Hadoop

Big Data With Hadoop With Saurabh Singh singh.903@osu.edu The Ohio State University February 11, 2016 Overview 1 2 3 Requirements Ecosystem Resilient Distributed Datasets (RDDs) Example Code vs Mapreduce 4 5 Source: [Tutorials

More information

Hybrid Cloud Storage System. Oh well, I will write the report on May1 st

Hybrid Cloud Storage System. Oh well, I will write the report on May1 st Project 2 Hybrid Cloud Storage System Project due on May 1 st (11.59 EST) Start early J : We have three graded milestones Milestone 1: demo part 1 by March 29 th Milestone 2: demo part 2 by April 12 th

More information

Storing Data: Disks and Files. Disks and Files. Why Not Store Everything in Main Memory? Chapter 7

Storing Data: Disks and Files. Disks and Files. Why Not Store Everything in Main Memory? Chapter 7 Storing : Disks and Files Chapter 7 Yea, from the table of my memory I ll wipe away all trivial fond records. -- Shakespeare, Hamlet base Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Disks and

More information

Google File System. Web and scalability

Google File System. Web and scalability Google File System Web and scalability The web: - How big is the Web right now? No one knows. - Number of pages that are crawled: o 100,000 pages in 1994 o 8 million pages in 2005 - Crawlable pages might

More information

CS 2112 Spring 2014. 0 Instructions. Assignment 3 Data Structures and Web Filtering. 0.1 Grading. 0.2 Partners. 0.3 Restrictions

CS 2112 Spring 2014. 0 Instructions. Assignment 3 Data Structures and Web Filtering. 0.1 Grading. 0.2 Partners. 0.3 Restrictions CS 2112 Spring 2014 Assignment 3 Data Structures and Web Filtering Due: March 4, 2014 11:59 PM Implementing spam blacklists and web filters requires matching candidate domain names and URLs very rapidly

More information

Practical Online Filesystem Checking and Repair

Practical Online Filesystem Checking and Repair Practical Online Filesystem Checking and Repair Daniel Phillips Samsung Research America (Silicon Valley) d.phillips@partner.samsung.com 1 2013 SAMSUNG Electronics Co. Why we want online checking: Fsck

More information

Lecture 18: Reliable Storage

Lecture 18: Reliable Storage CS 422/522 Design & Implementation of Operating Systems Lecture 18: Reliable Storage Zhong Shao Dept. of Computer Science Yale University Acknowledgement: some slides are taken from previous versions of

More information

Windows OS File Systems

Windows OS File Systems Windows OS File Systems MS-DOS and Windows 95/98/NT/2000/XP allow use of FAT-16 or FAT-32. Windows NT/2000/XP uses NTFS (NT File System) File Allocation Table (FAT) Not used so much, but look at as a contrast

More information

General Introduction

General Introduction Managed Runtime Technology: General Introduction Xiao-Feng Li (xiaofeng.li@gmail.com) 2012-10-10 Agenda Virtual machines Managed runtime systems EE and MM (JIT and GC) Summary 10/10/2012 Managed Runtime

More information

Best Practices on monitoring Solaris Global/Local Zones using IBM Tivoli Monitoring

Best Practices on monitoring Solaris Global/Local Zones using IBM Tivoli Monitoring Best Practices on monitoring Solaris Global/Local Zones using IBM Tivoli Monitoring Document version 1.0 Gianluca Della Corte, IBM Tivoli Monitoring software engineer Antonio Sgro, IBM Tivoli Monitoring

More information

Maximize MicroStrategy Speed and Throughput with High Performance Tuning

Maximize MicroStrategy Speed and Throughput with High Performance Tuning Maximize MicroStrategy Speed and Throughput with High Performance Tuning Jochen Demuth, Director Partner Engineering Maximize MicroStrategy Speed and Throughput with High Performance Tuning Agenda 1. Introduction

More information

Data storage Tree indexes

Data storage Tree indexes Data storage Tree indexes Rasmus Pagh February 7 lecture 1 Access paths For many database queries and updates, only a small fraction of the data needs to be accessed. Extreme examples are looking or updating

More information

Cloud Cruiser and Azure Public Rate Card API Integration

Cloud Cruiser and Azure Public Rate Card API Integration Cloud Cruiser and Azure Public Rate Card API Integration In this article: Introduction Azure Rate Card API Cloud Cruiser s Interface to Azure Rate Card API Import Data from the Azure Rate Card API Defining

More information

Computer Engineering and Systems Group Electrical and Computer Engineering SCMFS: A File System for Storage Class Memory

Computer Engineering and Systems Group Electrical and Computer Engineering SCMFS: A File System for Storage Class Memory SCMFS: A File System for Storage Class Memory Xiaojian Wu, Narasimha Reddy Texas A&M University What is SCM? Storage Class Memory Byte-addressable, like DRAM Non-volatile, persistent storage Example: Phase

More information

Chapter 3 Operating-System Structures

Chapter 3 Operating-System Structures Contents 1. Introduction 2. Computer-System Structures 3. Operating-System Structures 4. Processes 5. Threads 6. CPU Scheduling 7. Process Synchronization 8. Deadlocks 9. Memory Management 10. Virtual

More information

Lecture 16: Storage Devices

Lecture 16: Storage Devices CS 422/522 Design & Implementation of Operating Systems Lecture 16: Storage Devices Zhong Shao Dept. of Computer Science Yale University Acknowledgement: some slides are taken from previous versions of

More information

idedup: Latency-aware, inline data deduplication for primary storage

idedup: Latency-aware, inline data deduplication for primary storage idedup: Latency-aware, inline data deduplication for primary storage Kiran Srinivasan, Tim Bisson, Garth Goodson, Kaladhar Voruganti NetApp, Inc. {skiran, tbisson, goodson, kaladhar}@netapp.com Abstract

More information

Qlik Sense scalability

Qlik Sense scalability Qlik Sense scalability Visual analytics platform Qlik Sense is a visual analytics platform powered by an associative, in-memory data indexing engine. Based on users selections, calculations are computed

More information

On Benchmarking Popular File Systems

On Benchmarking Popular File Systems On Benchmarking Popular File Systems Matti Vanninen James Z. Wang Department of Computer Science Clemson University, Clemson, SC 2963 Emails: {mvannin, jzwang}@cs.clemson.edu Abstract In recent years,

More information

Speeding Up Cloud/Server Applications Using Flash Memory

Speeding Up Cloud/Server Applications Using Flash Memory Speeding Up Cloud/Server Applications Using Flash Memory Sudipta Sengupta Microsoft Research, Redmond, WA, USA Contains work that is joint with B. Debnath (Univ. of Minnesota) and J. Li (Microsoft Research,

More information

Near Real Time Indexing Kafka Message to Apache Blur using Spark Streaming. by Dibyendu Bhattacharya

Near Real Time Indexing Kafka Message to Apache Blur using Spark Streaming. by Dibyendu Bhattacharya Near Real Time Indexing Kafka Message to Apache Blur using Spark Streaming by Dibyendu Bhattacharya Pearson : What We Do? We are building a scalable, reliable cloud-based learning platform providing services

More information