Designing a Concurrent File Server

Transcription

1 Designing a Concurrent File Server James Whitehead II University of Oxford jim.whitehead@cs.ox.ac.uk 28 August 2012 Communicating Process Architectures 2012 Copyright 2012, James Whitehead II. 1 / 22

2 Building CSP Style Concurrent Systems Disciplined model of concurrency Build applications Web server (CPA 2011) File system server Range of designs Communicating Process Architectures 2012 Copyright 2012, James Whitehead II. 2 / 22

3 Go Programming Language Robert Griesemer, Rob Pike, Ken Thompson Fast compilation Lightweight type system Garbage collected Concurrent Communicating Process Architectures 2012 Copyright 2012, James Whitehead II. 3 / 22

4 Concurrency in Go Can FORK goroutines using go keyword No PAR/FORKING block Any-to-any channels ALT using select Cites CSP (but not occam) Don t communicate by sharing memory, share memory by communicating Communicating Process Architectures 2012 Copyright 2012, James Whitehead II. 4 / 22

5 File servers Contention for resources Low-level I/O Shared data (disk) Shared memory Multiple users Communicating Process Architectures 2012 Copyright 2012, James Whitehead II. 5 / 22

6 File server concurrency Highly concurrent with explicit locks (UNIX, Linux) No concurrency (MINIX) Simple source code Embarrassingly sequential MINIX provides a clean slate from which to derive the design of a concurrent file system. Communicating Process Architectures 2012 Copyright 2012, James Whitehead II. 6 / 22

7 Existing architecture proc proc proc system call interface FS Server syscall implementations inode table block cache message procedure call dev dev dev Communicating Process Architectures 2012 Copyright 2012, James Whitehead II. 7 / 22

8 Opportunities for concurrency Files must be opened for I/O File descriptors Available until closed read, write, seek, etc. An I/O operation concurrently with a normal system call. Communicating Process Architectures 2012 Copyright 2012, James Whitehead II. 8 / 22

9 A step in the right direction... proc proc proc system call interface I/O server FS server inode table alloc table block cache New I/O server Explicit dependencies All sequential processes Not free from hazards... Size of a file Updated in write (I/O) Read in stat (FS) Data hiding/segregation dev dev dev Communicating Process Architectures 2012 Copyright 2012, James Whitehead II. 9 / 22

10 Trickle-down concurrency Reduced the complexity of atomic actions, introducing three new bottlenecks. Block cache Inode table Alloc table Communicating Process Architectures 2012 Copyright 2012, James Whitehead II. 10 / 22

11 Block cache concurrency Cache-hit Return the block immediately Cache-miss Spawn new goroutine to load block Continue serving other requests Return block when ready Slowly but surely.. Communicating Process Architectures 2012 Copyright 2012, James Whitehead II. 11 / 22

12 Concurrent I/O Server Two open files are independent No shared blocks No shared inodes Allow CREW concurrency for each file Shared file descriptors must be addressed.. Communicating Process Architectures 2012 Copyright 2012, James Whitehead II. 12 / 22

13 A compromise... proc proc proc system call interface file I/O Server fd FS server file fd fd alloc table inode table block cache dev dev dev Communicating Process Architectures 2012 Copyright 2012, James Whitehead II. 13 / 22

14 Design Overall: Process encapsulation Eliminate implicit sharing Connections and interaction Easy to reason about Individual: Sequential is simple Concurrency applied carefully Communicating Process Architectures 2012 Copyright 2012, James Whitehead II. 14 / 22

15 A roadblock.. Concurrent file I/O Sequential (other) system calls Inodes are shared (disk, memory) Standards (under) specification Need to lock one or more inodes Communicating Process Architectures 2012 Copyright 2012, James Whitehead II. 15 / 22

16 Conclusions Compromise between sequential and highly concurrent Disciplined model of concurrency Iterative (and careful) introduction Techniques for addressing concurrency needs Communicating Process Architectures 2012 Copyright 2012, James Whitehead II. 16 / 22

17 A postlude on unit testing Communicating Process Architectures 2012 Copyright 2012, James Whitehead II. 17 / 22

18 Unit testing concurrency Want to show that a slow/broken device doesn t break the system, unit testing the concurrency properties. Create a broken device Use channels to order operations Interfaces and concurrency Communicating Process Architectures 2012 Copyright 2012, James Whitehead II. 18 / 22

19 Setup type BlockDevice interface { Read(buf interface{}, pos int64) error Write(buf interface{}, pos int64) error } type BlockingDevice struct { BlockDevice HasBlocked chan int64 // position argument Unblock chan bool } func (dev *BlockingDevice) Read(buf interface{}, pos int64) error { dev.hasblocked <- pos <-dev.unblock return dev.blockdevice.read(buf, pos) } Communicating Process Architectures 2012 Copyright 2012, James Whitehead II. 19 / 22

20 Blocking a device // create a block cache with test device, and a broken device cache := createtestcache() bdev := createblockingdevice() cache.mountdevice(1, bdev) // to join the spawned goroutines (manual PAR) done := make(chan bool) go func() { // do a read on the broken device (1) cb := cache.getblock(1, SUPER_BLOCK) cache.putblock(cb) done <- true }() Communicating Process Architectures 2012 Copyright 2012, James Whitehead II. 20 / 22

21 Testing the cache go func() { // wait for the device to be blocked <-bdev.hasblocked // then request a read from the non-broken device cb := cache.getblock(0, SUPER_BLOCK) // now release the broken device so it cleanly shuts down bdev.unblock <- true cache.putblock(cb) done <- true }() // wait for both goroutines to finish <-done <-done Communicating Process Architectures 2012 Copyright 2012, James Whitehead II. 21 / 22

22 Questions? Comments? Communicating Process Architectures 2012 Copyright 2012, James Whitehead II. 22 / 22