Distributed Systems [Fall 2012]

Size: px
Start display at page:

Download "Distributed Systems [Fall 2012]"

Transcription

1 Distributed Systems [Fall 2012] [W4995-2] Lec 6: YFS Lab Introduction and Local Synchronization Primitives Slide acks: Jinyang Li, Dave Andersen, Randy Bryant (http://www.news.cs.nyu.edu/~jinyang/fa10/notes/ds-lec2.ppt,

2 Outline Detailed YFS lab introduction Plus Lab 1 description From last lecture s slides, which we didn t cover Local synchronization primitives Semaphores Conditional variables Next time: distributed synchronization 2

3 YFS (Reminder) Instructional distributed file system developed by MIT after a research distributed file system, called Frangipani Analogous to xv6 for OS courses When we discuss YFS, we really refer to Frangipani (or a simplified version thereof) YFS/Frangipani Goals: Aggregate many disks from many s Incrementally scalable Tolerates and recovers from node, network, disk failures 3

4 Design (Reminder) client workstations (applications) file system interface (creat,read,write,..) YFS YFS YFS file service put/get acquire, release extent extent extent lock lock extent service (data) lock service 4

5 Design (Reminder) file system interface (creat,read,write,..) YFS extent put/get extent YFS extent lock YFS acquire, release Extent Service: stores the data (dir and file contents) replicated for fault tolerance incrementally scalable with more s lock YFS Service: serves file system requests it s stateless (i.e., doesn t store data) uses extent service to store data incrementally scalable with more s Lock Service: ensure consistent updates by multiple yfs s replicated for fault tolerance 5

6 YFS Server Implements FS Logic Application program: creat( /d1/f1, 0777) creat( /d1/f1 ) YFS YFS file system interface YFS put/get acquire, release YFS : extent extent extent lock lock 1. GET root directory s data from Extent Service 2. Find Extent Server address for dir /d1 in root dir s data 3. GET /d1 s data from Extent Server 4. Find Extent Server address of f1 in /d1 s data 5. If not exists alloc a new data block for f1 from Extent Service add f1 to /d1 s data, PUT modified /d1 to Extent Serv. 6

7 Concurrent Accesses Cause Inconsistency time App 1: creat( /d1/f1, 0777) Server S1: GET /d1 Find file f1 in /d1 If not exists PUT modified /d1 App 2: creat( /d1/f2, 0777) Server S2: GET /d1 Find file f2 in /d1 If not exists PUT modified /d1 What is the final result of /d1? What should it be? 7

8 Solution: Use a Lock Service to Synchronize Access time App 1: creat( /d1/f1, 0777) Server S1: LOCK( /d1 ) GET /d1 Find file f1 in /d1 If not exists PUT modified /d1 UNLOCK( /d1 ) App 2: creat( /d1/f2, 0777) Server S2: LOCK( /d1 ) GET /d1 8

9 Putting It Together creat( /d1/f1 ) YFS extent file system interface (creat,read,write,..) creat( /d1/f1 ) put/get YFS 2. get( /d1 ) 3. put( /d1 ) 4. release( /d1 ) extent extent 1. acquire( /d1 ) acquire, release lock lock YFS 9

10 Another Problem: Naming creat( /d1/f1 ) YFS extent file system interface (creat,read,write,..) creat( /d1/f1 ) 1. acquire( /d1 ) put/get extent YFS 2. get( /d1 ) 3. put( /d1 ) 4. release( /d1 ) extent acquire, release lock lock YFS Any issues with this version? 10

11 Another Problem: Naming time App 1: creat( /d0/d1/f1, 0777) Server S1: acquire( /d0/d1 ) GET /d1 Find file f1 in /d1 If not exists PUT modified /d1 release( /d0/d1 ) App 2: Server S2: rename( /d0, /d2 ) rename( /d3, /d0 ) Same problem occurs for reading/writing files, if we use their names when identifying them on the. 11

12 Solution: Using GUIDs time App 1: creat( /d0/d1/f1, 0777) Server S1: acquire(d1_id) d1 = GET d1_id Find file f1 in d1 If not exists PUT(d1_id, modified d1) release(d1_id) App 2: Server S2: rename( /d0, /d2 ) rename( /d3, /d0 ) GUIDs are globally-unique, location-independent names GUID principle is pervasive in FSes and DSes! Think of inode numbers 12

13 Labs 1-5: Build a Simplified YFS Version yfs yfs User-level kernel App FUSE syscall Single extent to store data Extent lock Communication using in-house RPC 13

14 Lab Schedule Lab 1: lock Programming with threads RPC semantics Lab 2: basic file Lab 3: sharing of files Lab 4: file locking Lab 5: caching locks 14

15 Lab 1: Lock Server Lock service consists of: Lock : grant a lock to clients, one at a time Lock client: talk to to acquire/release locks Correctness: At most one lock is granted to any client Additional requirement: acquire() at client does not return until lock is granted 15

16 Lab 1 Steps Step 1: Implement lock and client lock code Use in-house RPC mechanism: It s a bit more primitive than Thrift and the like (see next slide) Step 2: Implement at-most-once semantics How do we do that? Due next Friday, Sept 28 before midnight Absolutely no extensions! Lab is significantly more difficult than Lab 0, so start working on it NOW! Work in layers, submit imperfect but on time! 16

17 YFS s RPC library lock_client lock_ rpc call marshal args transmit RPC request receive marshal args rpc handler wait work rpc call return unmarshal args receive RPC response transmit unmarshal args rpc handler return rpcc rpcs cl->call(lock_protocol::acquire, x, ret).reg(lock_protocol::acquire, &ls, &lock_::acquire) 17

18 Outline Detailed YFS lab introduction Plus Lab 1 description From last lecture s slides, which we didn t cover Local synchronization primitives Semaphores Conditional variables Next time: distributed synchronization Many of the primitives here distribute or build upon local primitives 18

19 Thread Synchronization (Reminder) What s a thread? What s a race condition? How do we avoid race conditions? Locks are very low level, and often times you need more to accomplish a synchronization goal Hence, people have developed a variety of synchronization primitives that raise level of abstraction 19

20 Semaphores Integer variable x that allows interaction via 2 operations: x.p(): while (x == 0) wait; --x x.v(): ++x Both operations are done atomically All steps take place without any intervening operations Question: How do you implement a semaphore? When do we use semaphores? 20

21 Example: Thread-Safe FIFO Queue q.initialize(): initialize state q.insert(x): add item into queue q.remove(): block until queue not empty; return item at head of queue q.flush(): clear queue Assume that we already have a sequential queue implementation: sq But sq is not thread-safe! So, how do we make q thread-safe? 21

22 FIFO Queue with Mutexes q.initialize(): q.sq = NewSQueue() q.mutex = 1 q.insert(x): q.sq.insert(x) q.remove(): x = q.sq.remove() return x q.flush(): q.sq.flush() Are we done? 22

23 FIFO Queue with Mutexes q.initialize(): q.sq = NewSQueue() q.mutex = 1 q.insert(x): q.sq.insert(x) q.remove(): x = q.sq.remove() return x q.flush(): q.sq.flush() Are we done? Nope: Remove doesn t block when buffer s empty 23

24 FIFO Queue with Semaphores Use semaphore to count number of elements in queue q.initialize(): q.sq = NewSQueue() q.mutex = 1 q.items = 0 q.insert(x): q.sq.insert(x) q.items.v() q.remove(): q.items.p() x = q.sq.remove() return x q.flush(): q.sq.flush() q.items = 0 Are we done? 24

25 FIFO Queue with Semaphores Use semaphore to count number of elements in queue q.initialize(): q.sq = NewSQueue() q.mutex = 1 q.items = 0 q.insert(x): q.sq.insert(x) q.items.v() q.flush() q.remove(): q.items.p() x = q.sq.remove() return x q.flush(): q.sq.flush() q.items = 0 Are we done? Nope: Just Insert & Remove work fine, but Flush messes things up 25

26 Fixing Race with Mutex? q.initialize(): q.sq = NewSQueue() q.mutex = 1 q.items = 0 q.insert(x): q.sq.insert(x) q.items.v() q.remove(): q.items.p() x = q.sq.remove() return x q.flush(): q.sq.flush() q.items = 0 Are we done? Yes, from a correctness perspective Nope, from a liveness perspective -- deadlock 26

27 Condition Variables Condition variables provide synchronization point, where one thread suspends until activated by another Condition variable always associated with a mutex cvar.wait(): Must be called after locking mutex Atomically: { release mutex & suspend operation } When resume, lock the mutex (may have to wait for it) cvar.signal(): If no thread suspended, then no-op Wake up one suspended thread 27

28 FIFO Queue with Condition Variable q.initialize(): q.sq = NewSQueue() q.mutex = 1 q.cvar = NewCond(q.mutex) q.insert(x): q.sq.insert(x) q.cvar.signal() q.remove(): if q.sq.isempty(): q.cvar.wait() x = q.sq.remove() return x q.flush(): q.sq.flush() Are we done? Still Nope: Wait() has 3 steps: - Unlock atomic - Wait for signal q.flush() - Lock 28

29 Thread-Safe FIFO Queue q.initialize(): q.sq = NewSQueue() q.mutex = 1 q.cvar = NewCond(q.mutex) q.insert(x): q.sq.insert(x) q.cvar.signal() q.remove(): while q.sq.isempty(): q.cvar.wait() x = q.sq.remove() return x q.flush(): q.sq.flush() Actually, one could build this using mutexes Build semaphore using mutex Build cond variable using semaphore + mutex But, boy, it s a mind-bender to do that that s why you want higher level of abstraction 29

30 Synchronization in Distributed Systems As we ve already seen in YFS Lab, distributed systems have similar issues: Multiple processes on different machines share the same resource: the data (or a printer, or the user s screen, ) Synchronization is even hairier in distributed systems, as you have to worry about failures, not just overlappings E.g.: when you get a lock, you may not even know you ve gotten it 30

31 Analogy: The Generals Dilemma e.g., carrier pigeon Let s attack together at 0500pm OK Hmm, should I attack? Did he get my OK? Same with distributed systems: machines need to agree on how to progress via an unreliable medium Things can get even messier if generals/machines can also fail (or even worse, go rogue) 31

32 Distributed Synchronization Mechanisms Logical clocks: clock synchronization is a real issue in distributed systems, hence they often maintain logical clocks, which count operations on the shared resource Consensus: multiple machines reach majority agreement over the operations they should perform and their ordering Data consistency protocols: replicas evolve their states in predefined ways so as to reach a common state despite different views of the input Distributed locking services: machines grab locks from a centralized, but still distributed, locking service, so as to coordinate their accesses to shared resources (e.g., files) Distributed transactions: an operation that involves multiple services either succeeds or fails at all of them We re going to look at all of these in the following lectures 32

33 Next Time Clocks in distributed systems Clock synchronization problem Logical (protocol) clocks 33

Last Class: Semaphores

Last Class: Semaphores Last Class: Semaphores A semaphore S supports two atomic operations: S Wait(): get a semaphore, wait if busy semaphore S is available. S Signal(): release the semaphore, wake up a process if one is waiting

More information

We mean.network File System

We mean.network File System We mean.network File System Introduction: Remote File-systems When networking became widely available users wanting to share files had to log in across the net to a central machine This central machine

More information

Lab 2 : Basic File Server. Introduction

Lab 2 : Basic File Server. Introduction Lab 2 : Basic File Server Introduction In this lab, you will start your file system implementation by getting the following FUSE operations to work: CREATE/MKNOD, LOOKUP, and READDIR SETATTR, WRITE and

More information

Chapter 6, The Operating System Machine Level

Chapter 6, The Operating System Machine Level Chapter 6, The Operating System Machine Level 6.1 Virtual Memory 6.2 Virtual I/O Instructions 6.3 Virtual Instructions For Parallel Processing 6.4 Example Operating Systems 6.5 Summary Virtual Memory General

More information

Chapter 11 Distributed File Systems. Distributed File Systems

Chapter 11 Distributed File Systems. Distributed File Systems Chapter 11 Distributed File Systems Introduction Case studies NFS Coda 1 Distributed File Systems A distributed file system enables clients to access files stored on one or more remote file servers A file

More information

Network File System (NFS)

Network File System (NFS) Network File System (NFS) Brad Karp UCL Computer Science CS GZ03 / M030 10 th October 2011 NFS Is Relevant Original paper from 1985 Very successful, still widely used today Early result; much subsequent

More information

Client/Server and Distributed Computing

Client/Server and Distributed Computing Adapted from:operating Systems: Internals and Design Principles, 6/E William Stallings CS571 Fall 2010 Client/Server and Distributed Computing Dave Bremer Otago Polytechnic, N.Z. 2008, Prentice Hall Traditional

More information

COSC 6374 Parallel Computation. Parallel I/O (I) I/O basics. Concept of a clusters

COSC 6374 Parallel Computation. Parallel I/O (I) I/O basics. Concept of a clusters COSC 6374 Parallel Computation Parallel I/O (I) I/O basics Spring 2008 Concept of a clusters Processor 1 local disks Compute node message passing network administrative network Memory Processor 2 Network

More information

! Past few lectures: Ø Locks: provide mutual exclusion Ø Condition variables: provide conditional synchronization

! Past few lectures: Ø Locks: provide mutual exclusion Ø Condition variables: provide conditional synchronization 1 Synchronization Constructs! Synchronization Ø Coordinating execution of multiple threads that share data structures Semaphores and Monitors: High-level Synchronization Constructs! Past few lectures:

More information

Semaphores and Monitors: High-level Synchronization Constructs

Semaphores and Monitors: High-level Synchronization Constructs Semaphores and Monitors: High-level Synchronization Constructs 1 Synchronization Constructs Synchronization Coordinating execution of multiple threads that share data structures Past few lectures: Locks:

More information

1.4 SOFTWARE CONCEPTS

1.4 SOFTWARE CONCEPTS 22 INTRODUCTION CHAP. 1 1.4 SOFTWARE CONCEPTS Hardware for distributed systems is important, but it is software that largely determines what a distributed system actually looks like. Distributed systems

More information

CS11 Java. Fall 2014-2015 Lecture 7

CS11 Java. Fall 2014-2015 Lecture 7 CS11 Java Fall 2014-2015 Lecture 7 Today s Topics! All about Java Threads! Some Lab 7 tips Java Threading Recap! A program can use multiple threads to do several things at once " A thread can have local

More information

A Deduplication File System & Course Review

A Deduplication File System & Course Review A Deduplication File System & Course Review Kai Li 12/13/12 Topics A Deduplication File System Review 12/13/12 2 Traditional Data Center Storage Hierarchy Clients Network Server SAN Storage Remote mirror

More information

Network File System (NFS) Pradipta De pradipta.de@sunykorea.ac.kr

Network File System (NFS) Pradipta De pradipta.de@sunykorea.ac.kr Network File System (NFS) Pradipta De pradipta.de@sunykorea.ac.kr Today s Topic Network File System Type of Distributed file system NFS protocol NFS cache consistency issue CSE506: Ext Filesystem 2 NFS

More information

Amoeba Distributed Operating System

Amoeba Distributed Operating System Amoeba Distributed Operating System Matt Ramsay Tim Kiegel Heath Memmer CS470 Case Study Paper 4/19/02 Amoeba Introduction The Amoeba operating system began as a research project at Vrije Universiteit

More information

Web Email DNS Peer-to-peer systems (file sharing, CDNs, cycle sharing)

Web Email DNS Peer-to-peer systems (file sharing, CDNs, cycle sharing) 1 1 Distributed Systems What are distributed systems? How would you characterize them? Components of the system are located at networked computers Cooperate to provide some service No shared memory Communication

More information

CS3210: Crash consistency. Taesoo Kim

CS3210: Crash consistency. Taesoo Kim 1 CS3210: Crash consistency Taesoo Kim 2 Administrivia Quiz #2. Lab4-5, Ch 3-6 (read "xv6 book") Open laptop/book, no Internet 3:05pm ~ 4:25-30pm (sharp) NOTE Lab6: 10% bonus, a single lab (bump up your

More information

SYSTEM ecos Embedded Configurable Operating System

SYSTEM ecos Embedded Configurable Operating System BELONGS TO THE CYGNUS SOLUTIONS founded about 1989 initiative connected with an idea of free software ( commercial support for the free software ). Recently merged with RedHat. CYGNUS was also the original

More information

COSC 6374 Parallel Computation. Parallel I/O (I) I/O basics. Concept of a clusters

COSC 6374 Parallel Computation. Parallel I/O (I) I/O basics. Concept of a clusters COSC 6374 Parallel I/O (I) I/O basics Fall 2012 Concept of a clusters Processor 1 local disks Compute node message passing network administrative network Memory Processor 2 Network card 1 Network card

More information

Principles and characteristics of distributed systems and environments

Principles and characteristics of distributed systems and environments Principles and characteristics of distributed systems and environments Definition of a distributed system Distributed system is a collection of independent computers that appears to its users as a single

More information

Last Class: Communication in Distributed Systems. Today: Remote Procedure Calls

Last Class: Communication in Distributed Systems. Today: Remote Procedure Calls Last Class: Communication in Distributed Systems Structured or unstructured? Addressing? Blocking/non-blocking? Buffered or unbuffered? Reliable or unreliable? Server architecture Scalability Push or pull?

More information

Distributed File Systems

Distributed File Systems Distributed File Systems Paul Krzyzanowski Rutgers University October 28, 2012 1 Introduction The classic network file systems we examined, NFS, CIFS, AFS, Coda, were designed as client-server applications.

More information

Middleware Lou Somers

Middleware Lou Somers Middleware Lou Somers April 18, 2002 1 Contents Overview Definition, goals, requirements Four categories of middleware Transactional, message oriented, procedural, object Middleware examples XML-RPC, SOAP,

More information

Lesson Objectives. To provide a grand tour of the major operating systems components To provide coverage of basic computer system organization

Lesson Objectives. To provide a grand tour of the major operating systems components To provide coverage of basic computer system organization Lesson Objectives To provide a grand tour of the major operating systems components To provide coverage of basic computer system organization AE3B33OSD Lesson 1 / Page 2 What is an Operating System? A

More information

RAID Storage, Network File Systems, and DropBox

RAID Storage, Network File Systems, and DropBox RAID Storage, Network File Systems, and DropBox George Porter CSE 124 February 24, 2015 * Thanks to Dave Patterson and Hong Jiang Announcements Project 2 due by end of today Office hour today 2-3pm in

More information

G22.3250-001. Porcupine. Robert Grimm New York University

G22.3250-001. Porcupine. Robert Grimm New York University G22.3250-001 Porcupine Robert Grimm New York University Altogether Now: The Three Questions! What is the problem?! What is new or different?! What are the contributions and limitations? Porcupine from

More information

Distributed File Systems. Chapter 10

Distributed File Systems. Chapter 10 Distributed File Systems Chapter 10 Distributed File System a) A distributed file system is a file system that resides on different machines, but offers an integrated view of data stored on remote disks.

More information

W4118: semaphore and monitor. Instructor: Junfeng Yang

W4118: semaphore and monitor. Instructor: Junfeng Yang W4118: semaphore and monitor Instructor: Junfeng Yang Outline Semaphores Producer-consumer problem Monitors and condition variables 2 Semaphore motivation Problem with lock: ensures mutual exclusion, but

More information

Parallel Programming

Parallel Programming Parallel Programming 0024 Recitation Week 7 Spring Semester 2010 R 7 :: 1 0024 Spring 2010 Today s program Assignment 6 Review of semaphores Semaphores Semaphoren and (Java) monitors Semaphore implementation

More information

Linux Driver Devices. Why, When, Which, How?

Linux Driver Devices. Why, When, Which, How? Bertrand Mermet Sylvain Ract Linux Driver Devices. Why, When, Which, How? Since its creation in the early 1990 s Linux has been installed on millions of computers or embedded systems. These systems may

More information

I/O. Input/Output. Types of devices. Interface. Computer hardware

I/O. Input/Output. Types of devices. Interface. Computer hardware I/O Input/Output One of the functions of the OS, controlling the I/O devices Wide range in type and speed The OS is concerned with how the interface between the hardware and the user is made The goal in

More information

Agenda. Distributed System Structures. Why Distributed Systems? Motivation

Agenda. Distributed System Structures. Why Distributed Systems? Motivation Agenda Distributed System Structures CSCI 444/544 Operating Systems Fall 2008 Motivation Network structure Fundamental network services Sockets and ports Client/server model Remote Procedure Call (RPC)

More information

Client/Server Computing Distributed Processing, Client/Server, and Clusters

Client/Server Computing Distributed Processing, Client/Server, and Clusters Client/Server Computing Distributed Processing, Client/Server, and Clusters Chapter 13 Client machines are generally single-user PCs or workstations that provide a highly userfriendly interface to the

More information

Why Threads Are A Bad Idea (for most purposes)

Why Threads Are A Bad Idea (for most purposes) Why Threads Are A Bad Idea (for most purposes) John Ousterhout Sun Microsystems Laboratories john.ousterhout@eng.sun.com http://www.sunlabs.com/~ouster Introduction Threads: Grew up in OS world (processes).

More information

Microkernels & Database OSs. Recovery Management in QuickSilver. DB folks: Stonebraker81. Very different philosophies

Microkernels & Database OSs. Recovery Management in QuickSilver. DB folks: Stonebraker81. Very different philosophies Microkernels & Database OSs Recovery Management in QuickSilver. Haskin88: Roger Haskin, Yoni Malachi, Wayne Sawdon, Gregory Chan, ACM Trans. On Computer Systems, vol 6, no 1, Feb 1988. Stonebraker81 OS/FS

More information

Lecture 6: Semaphores and Monitors

Lecture 6: Semaphores and Monitors HW 2 Due Tuesday 10/18 Lecture 6: Semaphores and Monitors CSE 120: Principles of Operating Systems Alex C. Snoeren Higher-Level Synchronization We looked at using locks to provide mutual exclusion Locks

More information

Lecture 18: Reliable Storage

Lecture 18: Reliable Storage CS 422/522 Design & Implementation of Operating Systems Lecture 18: Reliable Storage Zhong Shao Dept. of Computer Science Yale University Acknowledgement: some slides are taken from previous versions of

More information

Monitors, Java, Threads and Processes

Monitors, Java, Threads and Processes Monitors, Java, Threads and Processes 185 An object-oriented view of shared memory A semaphore can be seen as a shared object accessible through two methods: wait and signal. The idea behind the concept

More information

First-class User Level Threads

First-class User Level Threads First-class User Level Threads based on paper: First-Class User Level Threads by Marsh, Scott, LeBlanc, and Markatos research paper, not merely an implementation report User-level Threads Threads managed

More information

Operating Systems 4 th Class

Operating Systems 4 th Class Operating Systems 4 th Class Lecture 1 Operating Systems Operating systems are essential part of any computer system. Therefore, a course in operating systems is an essential part of any computer science

More information

6.828 Operating System Engineering: Fall 2003. Quiz II Solutions THIS IS AN OPEN BOOK, OPEN NOTES QUIZ.

6.828 Operating System Engineering: Fall 2003. Quiz II Solutions THIS IS AN OPEN BOOK, OPEN NOTES QUIZ. Department of Electrical Engineering and Computer Science MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.828 Operating System Engineering: Fall 2003 Quiz II Solutions All problems are open-ended questions. In

More information

ZooKeeper. Table of contents

ZooKeeper. Table of contents by Table of contents 1 ZooKeeper: A Distributed Coordination Service for Distributed Applications... 2 1.1 Design Goals...2 1.2 Data model and the hierarchical namespace...3 1.3 Nodes and ephemeral nodes...

More information

Have both hardware and software. Want to hide the details from the programmer (user).

Have both hardware and software. Want to hide the details from the programmer (user). Input/Output Devices Chapter 5 of Tanenbaum. Have both hardware and software. Want to hide the details from the programmer (user). Ideally have the same interface to all devices (device independence).

More information

Linux Kernel Architecture

Linux Kernel Architecture Linux Kernel Architecture Amir Hossein Payberah payberah@yahoo.com Contents What is Kernel? Kernel Architecture Overview User Space Kernel Space Kernel Functional Overview File System Process Management

More information

Distributed Data Management

Distributed Data Management Introduction Distributed Data Management Involves the distribution of data and work among more than one machine in the network. Distributed computing is more broad than canonical client/server, in that

More information

Implementing the Hadoop Distributed File System Protocol on OneFS Jeff Hughes EMC Isilon

Implementing the Hadoop Distributed File System Protocol on OneFS Jeff Hughes EMC Isilon Implementing the Hadoop Distributed File System Protocol on OneFS Jeff Hughes EMC Isilon Outline Hadoop Overview OneFS Overview MapReduce + OneFS Details of isi_hdfs_d Wrap up & Questions 2 Hadoop Overview

More information

Dr Markus Hagenbuchner markus@uow.edu.au CSCI319. Distributed Systems

Dr Markus Hagenbuchner markus@uow.edu.au CSCI319. Distributed Systems Dr Markus Hagenbuchner markus@uow.edu.au CSCI319 Distributed Systems CSCI319 Chapter 8 Page: 1 of 61 Fault Tolerance Study objectives: Understand the role of fault tolerance in Distributed Systems. Know

More information

Real Time Programming: Concepts

Real Time Programming: Concepts Real Time Programming: Concepts Radek Pelánek Plan at first we will study basic concepts related to real time programming then we will have a look at specific programming languages and study how they realize

More information

Running a Workflow on a PowerCenter Grid

Running a Workflow on a PowerCenter Grid Running a Workflow on a PowerCenter Grid 2010-2014 Informatica Corporation. No part of this document may be reproduced or transmitted in any form, by any means (electronic, photocopying, recording or otherwise)

More information

Topics in Computer System Performance and Reliability: Storage Systems!

Topics in Computer System Performance and Reliability: Storage Systems! CSC 2233: Topics in Computer System Performance and Reliability: Storage Systems! Note: some of the slides in today s lecture are borrowed from a course taught by Greg Ganger and Garth Gibson at Carnegie

More information

E) Modeling Insights: Patterns and Anti-patterns

E) Modeling Insights: Patterns and Anti-patterns Murray Woodside, July 2002 Techniques for Deriving Performance Models from Software Designs Murray Woodside Second Part Outline ) Conceptual framework and scenarios ) Layered systems and models C) uilding

More information

Processes and Non-Preemptive Scheduling. Otto J. Anshus

Processes and Non-Preemptive Scheduling. Otto J. Anshus Processes and Non-Preemptive Scheduling Otto J. Anshus 1 Concurrency and Process Challenge: Physical reality is Concurrent Smart to do concurrent software instead of sequential? At least we want to have

More information

Review from last time. CS 537 Lecture 3 OS Structure. OS structure. What you should learn from this lecture

Review from last time. CS 537 Lecture 3 OS Structure. OS structure. What you should learn from this lecture Review from last time CS 537 Lecture 3 OS Structure What HW structures are used by the OS? What is a system call? Michael Swift Remzi Arpaci-Dussea, Michael Swift 1 Remzi Arpaci-Dussea, Michael Swift 2

More information

CSE 120 Principles of Operating Systems. Modules, Interfaces, Structure

CSE 120 Principles of Operating Systems. Modules, Interfaces, Structure CSE 120 Principles of Operating Systems Fall 2000 Lecture 3: Operating System Modules, Interfaces, and Structure Geoffrey M. Voelker Modules, Interfaces, Structure We roughly defined an OS as the layer

More information

Chapter 18: Database System Architectures. Centralized Systems

Chapter 18: Database System Architectures. Centralized Systems Chapter 18: Database System Architectures! Centralized Systems! Client--Server Systems! Parallel Systems! Distributed Systems! Network Types 18.1 Centralized Systems! Run on a single computer system and

More information

A distributed system is defined as

A distributed system is defined as A distributed system is defined as A collection of independent computers that appears to its users as a single coherent system CS550: Advanced Operating Systems 2 Resource sharing Openness Concurrency

More information

Network Attached Storage. Jinfeng Yang Oct/19/2015

Network Attached Storage. Jinfeng Yang Oct/19/2015 Network Attached Storage Jinfeng Yang Oct/19/2015 Outline Part A 1. What is the Network Attached Storage (NAS)? 2. What are the applications of NAS? 3. The benefits of NAS. 4. NAS s performance (Reliability

More information

Middleware: Past and Present a Comparison

Middleware: Past and Present a Comparison Middleware: Past and Present a Comparison Hennadiy Pinus ABSTRACT The construction of distributed systems is a difficult task for programmers, which can be simplified with the use of middleware. Middleware

More information

Introduction. What is an Operating System?

Introduction. What is an Operating System? Introduction What is an Operating System? 1 What is an Operating System? 2 Why is an Operating System Needed? 3 How Did They Develop? Historical Approach Affect of Architecture 4 Efficient Utilization

More information

SSC - Concurrency and Multi-threading Java multithreading programming - Synchronisation (I)

SSC - Concurrency and Multi-threading Java multithreading programming - Synchronisation (I) SSC - Concurrency and Multi-threading Java multithreading programming - Synchronisation (I) Shan He School for Computational Science University of Birmingham Module 06-19321: SSC Outline Outline of Topics

More information

Lecture 6: Introduction to Monitors and Semaphores

Lecture 6: Introduction to Monitors and Semaphores Concurrent Programming 19530-V (WS01) Lecture 6: Introduction to Monitors and Semaphores Dr. Richard S. Hall rickhall@inf.fu-berlin.de Concurrent programming November 27, 2001 Abstracting Locking Details

More information

The ConTract Model. Helmut Wächter, Andreas Reuter. November 9, 1999

The ConTract Model. Helmut Wächter, Andreas Reuter. November 9, 1999 The ConTract Model Helmut Wächter, Andreas Reuter November 9, 1999 Overview In Ahmed K. Elmagarmid: Database Transaction Models for Advanced Applications First in Andreas Reuter: ConTracts: A Means for

More information

Centralized Systems. A Centralized Computer System. Chapter 18: Database System Architectures

Centralized Systems. A Centralized Computer System. Chapter 18: Database System Architectures Chapter 18: Database System Architectures Centralized Systems! Centralized Systems! Client--Server Systems! Parallel Systems! Distributed Systems! Network Types! Run on a single computer system and do

More information

Lecture 25 Symbian OS

Lecture 25 Symbian OS CS 423 Operating Systems Design Lecture 25 Symbian OS Klara Nahrstedt Fall 2011 Based on slides from Andrew S. Tanenbaum textbook and other web-material (see acknowledgements) cs423 Fall 2011 1 Overview

More information

Big Data Storage Options for Hadoop Sam Fineberg, HP Storage

Big Data Storage Options for Hadoop Sam Fineberg, HP Storage Sam Fineberg, HP Storage SNIA Legal Notice The material contained in this tutorial is copyrighted by the SNIA unless otherwise noted. Member companies and individual members may use this material in presentations

More information

Database Replication with Oracle 11g and MS SQL Server 2008

Database Replication with Oracle 11g and MS SQL Server 2008 Database Replication with Oracle 11g and MS SQL Server 2008 Flavio Bolfing Software and Systems University of Applied Sciences Chur, Switzerland www.hsr.ch/mse Abstract Database replication is used widely

More information

Chapter 17: Distributed-File Systems. Operating System Concepts 8 th Edition,

Chapter 17: Distributed-File Systems. Operating System Concepts 8 th Edition, Chapter 17: Distributed-File Systems, Silberschatz, Galvin and Gagne 2009 Chapter 17 Distributed-File Systems Background Naming and Transparency Remote File Access Stateful versus Stateless Service File

More information

Operating Systems Principles

Operating Systems Principles bicfm page i Operating Systems Principles Lubomir F. Bic University of California, Irvine Alan C. Shaw University of Washington, Seattle PEARSON EDUCATION INC. Upper Saddle River, New Jersey 07458 bicfm

More information

File Management. COMP3231 Operating Systems. Kevin Elphinstone. Tanenbaum, Chapter 4

File Management. COMP3231 Operating Systems. Kevin Elphinstone. Tanenbaum, Chapter 4 File Management Tanenbaum, Chapter 4 COMP3231 Operating Systems Kevin Elphinstone 1 Outline Files and directories from the programmer (and user) perspective Files and directories internals the operating

More information

Local Area Networks transmission system private speedy and secure kilometres shared transmission medium hardware & software

Local Area Networks transmission system private speedy and secure kilometres shared transmission medium hardware & software Local Area What s a LAN? A transmission system, usually private owned, very speedy and secure, covering a geographical area in the range of kilometres, comprising a shared transmission medium and a set

More information

CS161: Operating Systems

CS161: Operating Systems CS161: Operating Systems Matt Welsh mdw@eecs.harvard.edu Lecture 18: RAID April 19, 2007 2007 Matt Welsh Harvard University 1 RAID Redundant Arrays of Inexpensive Disks Invented in 1986-1987 by David Patterson

More information

A Comparison Of Shared Memory Parallel Programming Models. Jace A Mogill David Haglin

A Comparison Of Shared Memory Parallel Programming Models. Jace A Mogill David Haglin A Comparison Of Shared Memory Parallel Programming Models Jace A Mogill David Haglin 1 Parallel Programming Gap Not many innovations... Memory semantics unchanged for over 50 years 2010 Multi-Core x86

More information

Distribution transparency. Degree of transparency. Openness of distributed systems

Distribution transparency. Degree of transparency. Openness of distributed systems Distributed Systems Principles and Paradigms Maarten van Steen VU Amsterdam, Dept. Computer Science steen@cs.vu.nl Chapter 01: Version: August 27, 2012 1 / 28 Distributed System: Definition A distributed

More information

Unit 2 Distributed Systems R.Yamini Dept. Of CA, SRM University Kattankulathur

Unit 2 Distributed Systems R.Yamini Dept. Of CA, SRM University Kattankulathur Unit 2 Distributed Systems R.Yamini Dept. Of CA, SRM University Kattankulathur 1 Introduction to Distributed Systems Why do we develop distributed systems? availability of powerful yet cheap microprocessors

More information

Chapter 11 I/O Management and Disk Scheduling

Chapter 11 I/O Management and Disk Scheduling Operating Systems: Internals and Design Principles, 6/E William Stallings Chapter 11 I/O Management and Disk Scheduling Dave Bremer Otago Polytechnic, NZ 2008, Prentice Hall I/O Devices Roadmap Organization

More information

Distributed Operating Systems Introduction

Distributed Operating Systems Introduction Distributed Operating Systems Introduction Ewa Niewiadomska-Szynkiewicz and Adam Kozakiewicz ens@ia.pw.edu.pl, akozakie@ia.pw.edu.pl Institute of Control and Computation Engineering Warsaw University of

More information

It is the thinnest layer in the OSI model. At the time the model was formulated, it was not clear that a session layer was needed.

It is the thinnest layer in the OSI model. At the time the model was formulated, it was not clear that a session layer was needed. Session Layer The session layer resides above the transport layer, and provides value added services to the underlying transport layer services. The session layer (along with the presentation layer) add

More information

CS 377: Operating Systems. Outline. A review of what you ve learned, and how it applies to a real operating system. Lecture 25 - Linux Case Study

CS 377: Operating Systems. Outline. A review of what you ve learned, and how it applies to a real operating system. Lecture 25 - Linux Case Study CS 377: Operating Systems Lecture 25 - Linux Case Study Guest Lecturer: Tim Wood Outline Linux History Design Principles System Overview Process Scheduling Memory Management File Systems A review of what

More information

A Flexible Scheduling Framework (for Linux): Supporting Multiple Programming Models with Arbitrary Semantics

A Flexible Scheduling Framework (for Linux): Supporting Multiple Programming Models with Arbitrary Semantics A Flexible Scheduling Framework (for Linux): Supporting Multiple Programming Models with Arbitrary Semantics Noah Watkins, Jared Straub*, Douglas Niehaus* Presented by Noah Watkins Systems Research Lab

More information

Operating Systems. Distributed Systems. A distributed system is:

Operating Systems. Distributed Systems. A distributed system is: Operating Systems Interface between the hardware and the rest: editors, compilers, database systems, application programs, your programs, etc. Allows portability, enables easier programming, The manager

More information

Project Group High- performance Flexible File System

Project Group High- performance Flexible File System Project Group High- performance Flexible File System Lecture 3: Network File Systems André Brinkmann Example: SUN NFS NFS (Network File System) is an open protocol to exchange files Developed by Sun Microsystems

More information

COS 318: Operating Systems

COS 318: Operating Systems COS 318: Operating Systems File Performance and Reliability Andy Bavier Computer Science Department Princeton University http://www.cs.princeton.edu/courses/archive/fall10/cos318/ Topics File buffer cache

More information

COMP5426 Parallel and Distributed Computing. Distributed Systems: Client/Server and Clusters

COMP5426 Parallel and Distributed Computing. Distributed Systems: Client/Server and Clusters COMP5426 Parallel and Distributed Computing Distributed Systems: Client/Server and Clusters Client/Server Computing Client Client machines are generally single-user workstations providing a user-friendly

More information

COS 318: Operating Systems. Virtual Machine Monitors

COS 318: Operating Systems. Virtual Machine Monitors COS 318: Operating Systems Virtual Machine Monitors Andy Bavier Computer Science Department Princeton University http://www.cs.princeton.edu/courses/archive/fall10/cos318/ Introduction Have been around

More information

Last Class: OS and Computer Architecture. Last Class: OS and Computer Architecture

Last Class: OS and Computer Architecture. Last Class: OS and Computer Architecture Last Class: OS and Computer Architecture System bus Network card CPU, memory, I/O devices, network card, system bus Lecture 3, page 1 Last Class: OS and Computer Architecture OS Service Protection Interrupts

More information

Distributed Systems: Concepts and Design

Distributed Systems: Concepts and Design Distributed Systems: Concepts and Design Edition 3 By George Coulouris, Jean Dollimore and Tim Kindberg Addison-Wesley, Pearson Education 2001. Chapter 2 Exercise Solutions 2.1 Describe and illustrate

More information

Real-Time Systems Prof. Dr. Rajib Mall Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur

Real-Time Systems Prof. Dr. Rajib Mall Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Real-Time Systems Prof. Dr. Rajib Mall Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Lecture No. # 26 Real - Time POSIX. (Contd.) Ok Good morning, so let us get

More information

CSC 2405: Computer Systems II

CSC 2405: Computer Systems II CSC 2405: Computer Systems II Spring 2013 (TR 8:30-9:45 in G86) Mirela Damian http://www.csc.villanova.edu/~mdamian/csc2405/ Introductions Mirela Damian Room 167A in the Mendel Science Building mirela.damian@villanova.edu

More information

10.2 THE CODA FILE SYSTEM

10.2 THE CODA FILE SYSTEM 604 DISTRIBUTED FILE SYSTEMS CHAP. 10 10.2 THE CODA FILE SYSTEM Our next example of a distributed file system is Coda. Coda has been developed at Carnegie Mellon University (CMU) in the 1990s, and is now

More information

The Microsoft Windows Hypervisor High Level Architecture

The Microsoft Windows Hypervisor High Level Architecture The Microsoft Windows Hypervisor High Level Architecture September 21, 2007 Abstract The Microsoft Windows hypervisor brings new virtualization capabilities to the Windows Server operating system. Its

More information

CSE 544 Principles of Database Management Systems. Magdalena Balazinska Fall 2007 Lecture 5 - DBMS Architecture

CSE 544 Principles of Database Management Systems. Magdalena Balazinska Fall 2007 Lecture 5 - DBMS Architecture CSE 544 Principles of Database Management Systems Magdalena Balazinska Fall 2007 Lecture 5 - DBMS Architecture References Anatomy of a database system. J. Hellerstein and M. Stonebraker. In Red Book (4th

More information

Chapter 13 File and Database Systems

Chapter 13 File and Database Systems Chapter 13 File and Database Systems Outline 13.1 Introduction 13.2 Data Hierarchy 13.3 Files 13.4 File Systems 13.4.1 Directories 13.4. Metadata 13.4. Mounting 13.5 File Organization 13.6 File Allocation

More information

Chapter 13 File and Database Systems

Chapter 13 File and Database Systems Chapter 13 File and Database Systems Outline 13.1 Introduction 13.2 Data Hierarchy 13.3 Files 13.4 File Systems 13.4.1 Directories 13.4. Metadata 13.4. Mounting 13.5 File Organization 13.6 File Allocation

More information

DISTRIBUTED AND PARALLELL DATABASE

DISTRIBUTED AND PARALLELL DATABASE DISTRIBUTED AND PARALLELL DATABASE SYSTEMS Tore Risch Uppsala Database Laboratory Department of Information Technology Uppsala University Sweden http://user.it.uu.se/~torer PAGE 1 What is a Distributed

More information

CHAPTER 6: DISTRIBUTED FILE SYSTEMS

CHAPTER 6: DISTRIBUTED FILE SYSTEMS CHAPTER 6: DISTRIBUTED FILE SYSTEMS Chapter outline DFS design and implementation issues: system structure, access, and sharing semantics Transaction and concurrency control: serializability and concurrency

More information

1 Organization of Operating Systems

1 Organization of Operating Systems COMP 730 (242) Class Notes Section 10: Organization of Operating Systems 1 Organization of Operating Systems We have studied in detail the organization of Xinu. Naturally, this organization is far from

More information

Distributed File Systems. NFS Architecture (1)

Distributed File Systems. NFS Architecture (1) COP 6611 Advanced Operating System Distributed File Systems Chi Zhang czhang@cs.fiu.edu NFS Architecture (1) a) The remote access model. (like NFS) b) The upload/download model (like FTP) 2 1 NFS Architecture

More information

Chapter 2: Remote Procedure Call (RPC)

Chapter 2: Remote Procedure Call (RPC) Chapter 2: Remote Procedure Call (RPC) Gustavo Alonso Computer Science Department Swiss Federal Institute of Technology (ETHZ) alonso@inf.ethz.ch http://www.iks.inf.ethz.ch/ Contents - Chapter 2 - RPC

More information

Distributed Systems LEEC (2005/06 2º Sem.)

Distributed Systems LEEC (2005/06 2º Sem.) Distributed Systems LEEC (2005/06 2º Sem.) Introduction João Paulo Carvalho Universidade Técnica de Lisboa / Instituto Superior Técnico Outline Definition of a Distributed System Goals Connecting Users

More information

Operating system Dr. Shroouq J.

Operating system Dr. Shroouq J. 3 OPERATING SYSTEM STRUCTURES An operating system provides the environment within which programs are executed. The design of a new operating system is a major task. The goals of the system must be well

More information