MINIX3: A Reliable and Secure Operating System Andrew S. Tanenbaum and a team of students and programmers who actually did all the work Vrije Universiteit Amsterdam, The Netherlands 1
GOAL OF OUR WORK: BUILD A RELIABLE OS Tanenbaum s definition of a reliable OS: An operating system is said to be reliable when a typical user has never experienced even a single failure in his or her lifetime and does not know anybody who has ever experienced a failure. In engineering terms, this is probably mean time to failure > 50 years I don t think we are there yet 2
THE TELEVISION MODEL 1. You buy the television 2. You plug it in 3. It works perfectly for the next 10 years 3
THE COMPUTER MODEL (WINDOWS EDITION) 1. You buy the computer 2. You plug it in 3. You install service packs 1 through 9f 4. You install 18 new emergency security patches 5. You find and install 7 new device drivers 6. You install antivirus software 7. You install antispyware software 8. You install antihacker software (firewall) 9. You install antispam software 10. You reboot the computer 4
THE COMPUTER MODEL (2) 11. It doesn t work 12. You call the helpdesk 13. You wait on hold for 30 minutes 14. They tell you to reinstall Windows 5
TYPICAL USER REACTION The New York Times recently reported that 25% of computer users have gotten so angry at their computer that they physically hit it. 6
IS RELIABILITY SO IMPORTANT? Annoying Lost work But also think about Industrial control systems in factories Power grids Hospital operating rooms Banking and e-commerce servers Emergency phone centers Control software in cars, airplanes, etc. 7
IS THIS FEASIBLE? We won t find out if we don t try Dutch Royal Academy gave me 2 million to try European Union gave me 2.5 million to give it a shot So, we re trying 8
IS RELIABILITY ACHIEVABLE AT ALL? Systems can survive hardware failures! RAIDs can survive failed disks ECC memory can survive parity errors in memory TCP/IP can survive lost packets CD-ROM drives can correct many simultaneous errors We need to be able to survive software failures, too 9
A NEED TO RETHINK OPERATING SYSTEMS Operating systems research need to be refocused We have nearly infinite hardware on PC-class machines Plenty of CPU cycles, RAM, bandwidth Current software has tons of (useless) features Consequently, the software is slow, bloated, and buggy To achieve the TV model, future OSes, must be Small Simple Modular Reliable Secure Self-healing 10
BRIEF HISTORY OF OUR WORK (1976) John Lions wrote a book on UNIX V6 (1979) AT&T released V7 and forbade books on it L (1985) I started to write a UNIX-like OS from scratch (1987) MINIX 1 + book for teaching OS classes released (1997) MINIX 2 (POSIX) & 2 nd edition of book released (2000) MINIX 2 license changed to BSD (2004) MINIX 3: start of work making a reliable OS (2006) 3 rd edition of book (2008) European grant (2010) Focus moved towards embedded systems (2013) MINIX 3.3.0 moves to NetBSD compatibility 11
THREE EDITIONS OF THE BOOK 1 2 3 12
INTELLIGENT DESIGN AS APPLIED TO OPERATING SYSTEMS Microkernel (15,000 LoC vs. > 15 million for Linux) Bugs per 1000 LoC: Most S/W (1-10) MINIX 3 at least 15 kernel bugs; Linux has > 15,000 Drivers have 3-7x more bugs than rest of kernel About 70% of the code is drivers Highly modular OS runs as multiple user-mode server processes 13
STEP 1: ISOLATE COMPONENTS Move all loadable modules out of the kernel includes all device drivers and file systems Run each module as a separate process with POLA (Principle Of Least Authority) 14
STEP 2: ISOLATE I/O Isolate I/O devices Limit access to I/O ports Constrain DMA (needs hardware assistance) 15
STEP 3: ISOLATE COMMUNICATION Limit interprocess communication Restrict kernel calls on a per component basis Restrict IPC on a need-to-communicate basis Make sure faulty receiver cannot hang sender 16
ARCHITECTURE OF MINIX 3 Process Shell Make... User User mode FS 1 FS 2 Proc.... Other Servers Disk TTY Net Print... Other Drivers Kernel mode Microkernel handles interrupts, processes, scheduling, IPC 17
USER-MODE DEVICE DRIVERS Each driver runs as a user-mode process No superuser privileges Protected by the MMU Do not have access to I/O ports, privileged instrs 18
USER-MODE SERVERS Each server runs as a separate process Some key servers Virtual file server Actual file servers Process manager Memory manager Network server Reincarnation server 19
A SIMPLIFIED EXAMPLE: DOING A READ 1 User Users User mode FS 4 Servers Disk 2 3 Drivers Kernel File access when the block is in the FS cache 20
FILE SERVER (2) 1 User Users User mode 2 FS 9 6 Servers 5 3 Disk 4 7,8 Drivers Notification Kernel File access when the block is NOT in the FS cache 21
REINCARNATION SERVER Parent of all the drivers and servers When a driver or server dies, RS collects it RS checks a table for action to take e.g., restart it RS also pings drivers and servers frequently 22
DISK DRIVER RECOVERY RS 1 User Users User mode 4 5 2 FS Servers New driver Disk driver X 3. Crash! Drivers Kernel System is self healing this is how we hope to make it reliable 23
KERNEL RELIABILITY/SECURITY Fewer LoC means fewer kernel bugs Small kernel (15,000 LoC) means reduced TCB NO foreign code (e.g., drivers) in the kernel Static data structures (no malloc in kernel) Moving bugs to user space reduces their power 24
IPC RELIABILITY/SECURITY Fixed-length messages (no buffer overruns) Rendezvous system was simple No lost messages No buffer management We had to add asynchronous messages Interrupts and messages are unified 25
DRIVER RELIABILITY/SECURITY Untrusted code: heavily isolated Bugs, viruses cannot spread to other modules Cannot touch kernel data structures Bad pointers crash only one driver; recoverable Infinite loops detected and driver restarted Restricted power to do damage (not superuser) 26
OTHER ADVANTAGES OF USER DRIVERS Short development cycle Normal programming model No down time for crash and reboot Easy debugging Good flexibility 27
FAULT INJECTION EXPERIMENT We injected 800,000 faults into each of 3 drivers Done on the binary drivers Examples, change src addr, dest addr, loop condition 100 faults were injected on each experiment Waited 1 sec to see if the driver crashed If no crash, inject another 100 faults and repeat The driver crashed in 18,038 trials The operating system NEVER crashed 28
PORT OF MINIX 3 TO ARM Restructured source tree for multiple architectures Changed booting to support uboot for ARM Rewrote the low-level code dealing with hardware Changed code for context switching, paging, etc. Removed x86 segmentation code Imported NetBSD ARM headers and libraries Ported build.sh for cross-toolchain support Wrote drivers for SD card and other Beagle devices 29
EMBEDDED SYSTEMS 5 cm BeagleBone Black 9 cm 30
CHARACTERISTICS Item Beaglebone Black 31
CHARACTERISTICS CPU Item Beaglebone Black ARM v7 32
CHARACTERISTICS CPU Item Clock Beaglebone Black ARM v7 1 GHz 33
CHARACTERISTICS Item CPU Clock RAM Beaglebone Black ARM v7 1 GHz 512 MB 34
CHARACTERISTICS Item CPU Clock RAM Flash Beaglebone Black ARM v7 1 GHz 512 MB 4 GB 35
CHARACTERISTICS Item CPU Clock RAM Flash Video Beaglebone Black ARM v7 1 GHz 512 MB 4 GB HDMI/1080p 36
CHARACTERISTICS Item Beaglebone Black CPU ARM v7 Clock 1 GHz RAM 512 MB Flash 4 GB Video HDMI/1080p GPIO pins 92 37
CHARACTERISTICS Item Beaglebone Black CPU ARM v7 Clock 1 GHz RAM 512 MB Flash 4 GB Video HDMI/1080p GPIO pins 92 Ethernet 10/100 Mbps 38
CHARACTERISTICS Item Beaglebone Black CPU ARM v7 Clock 1 GHz RAM 512 MB Flash 4 GB Video HDMI/1080p GPIO pins 92 Ethernet 10/100 Mbps USB 1 39
CHARACTERISTICS Item Beaglebone Black CPU ARM v7 Clock 1 GHz RAM 512 MB Flash 4 GB Video HDMI/1080p GPIO pins 92 Ethernet 10/100 Mbps USB 1 Open source Yes 40
CHARACTERISTICS Item Beaglebone Black CPU ARM v7 Clock 1 GHz RAM 512 MB Flash 4 GB Video HDMI/1080p GPIO pins 92 Ethernet 10/100 Mbps USB 1 Open source Yes Price (quantity 1) $45 41
CHARACTERISTICS Item Beaglebone Black Raspberry Pi B+ CPU ARM v7 Clock 1 GHz RAM 512 MB Flash 4 GB Video HDMI/1080p GPIO pins 92 Ethernet 10/100 Mbps USB 1 Open source Yes Price (quantity 1) $45 42
CHARACTERISTICS Item Beaglebone Black Raspberry Pi B+ CPU ARM v7 ARM v6 Clock 1 GHz RAM 512 MB Flash 4 GB Video HDMI/1080p GPIO pins 92 Ethernet 10/100 Mbps USB 1 Open source Yes Price (quantity 1) $45 43
CHARACTERISTICS Item Beaglebone Black Raspberry Pi B+ CPU ARM v7 ARM v6 Clock 1 GHz 700 MHz RAM 512 MB Flash 4 GB Video HDMI/1080p GPIO pins 92 Ethernet 10/100 Mbps USB 1 Open source Yes Price (quantity 1) $45 44
CHARACTERISTICS Item Beaglebone Black Raspberry Pi B+ CPU ARM v7 ARM v6 Clock 1 GHz 700 MHz RAM 512 MB 512 MB Flash 4 GB Video HDMI/1080p GPIO pins 92 Ethernet 10/100 Mbps USB 1 Open source Yes Price (quantity 1) $45 45
CHARACTERISTICS Item Beaglebone Black Raspberry Pi B+ CPU ARM v7 ARM v6 Clock 1 GHz 700 MHz RAM 512 MB 512 MB Flash 4 GB None Video HDMI/1080p GPIO pins 92 Ethernet 10/100 Mbps USB 1 Open source Yes Price (quantity 1) $45 46
CHARACTERISTICS Item Beaglebone Black Raspberry Pi B+ CPU ARM v7 ARM v6 Clock 1 GHz 700 MHz RAM 512 MB 512 MB Flash 4 GB None Video HDMI/1080p HDMI/1080p GPIO pins 92 Ethernet 10/100 Mbps USB 1 Open source Yes Price (quantity 1) $45 47
CHARACTERISTICS Item Beaglebone Black Raspberry Pi B+ CPU ARM v7 ARM v6 Clock 1 GHz 700 MHz RAM 512 MB 512 MB Flash 4 GB None Video HDMI/1080p HDMI/1080p GPIO pins 92 40 Ethernet 10/100 Mbps USB 1 Open source Yes Price (quantity 1) $45 48
CHARACTERISTICS Item Beaglebone Black Raspberry Pi B+ CPU ARM v7 ARM v6 Clock 1 GHz 700 MHz RAM 512 MB 512 MB Flash 4 GB None Video HDMI/1080p HDMI/1080p GPIO pins 92 40 Ethernet 10/100 Mbps 10/100 Mbps USB 1 Open source Yes Price (quantity 1) $45 49
CHARACTERISTICS Item Beaglebone Black Raspberry Pi B+ CPU ARM v7 ARM v6 Clock 1 GHz 700 MHz RAM 512 MB 512 MB Flash 4 GB None Video HDMI/1080p HDMI/1080p GPIO pins 92 40 Ethernet 10/100 Mbps 10/100 Mbps USB 1 4 Open source Yes Price (quantity 1) $45 50
CHARACTERISTICS Item Beaglebone Black Raspberry Pi B+ CPU ARM v7 ARM v6 Clock 1 GHz 700 MHz RAM 512 MB 512 MB Flash 4 GB None Video HDMI/1080p HDMI/1080p GPIO pins 92 40 Ethernet 10/100 Mbps 10/100 Mbps USB 1 4 Open source Yes No Price (quantity 1) $45 51
CHARACTERISTICS Item Beaglebone Black Raspberry Pi B+ CPU ARM v7 ARM v6 Clock 1 GHz 700 MHz RAM 512 MB 512 MB Flash 4 GB None Video HDMI/1080p HDMI/1080p GPIO pins 92 40 Ethernet 10/100 Mbps 10/100 Mbps USB 1 4 Open source Yes No Price (quantity 1) $45 $35 52
I ADMIT I WAS WRONG On 29 Jan 1992 I posted to comp.os.minix this: Don`t get me wrong, I am not unhappy with LINUX. It will get all the people who want to turn MINIX in BSD UNIX off my back. I Apologize. Now I do want to turn MINIX into BSD. It just took me 20 years to realize this. 53
MINIX 3 MEETS BSD + = BSD Daemon is copyright 1988 by Marshall Kirk McKusick and is used with permission. 54
OR MAYBE 55
WHY BSD? MINIX 3 didn t have enough application software BSD is a proven, portable, quality product BSD has better code quality than Linux Pkgsrc handles packages better than what we had Thousands of excellent packages available Active community License compatibility Why NetBSD? Mostly due to its emphasis on portability 56
NETBSD FEATURES IN MINIX 3.3.0 Clang/LLVM compiler NetBSD build system ELF file format Source code tree modeled on NetBSD Headers and libraries are from NetBSD X11 Pkgsrc works and builds 5040 NetBSD packages Nevertheless, it is built on MINIX 3 kernel & servers 57
NETBSD FEATURES MISSING IN MINIX 3.3.0 Kernel threads (we do have userland pthreads) Some system calls: All _LWP*, MSG*, SEM* calls CLONE Some GET, IOCTL calls KQUEUE, KTRACE VFORK Job control Some other minor calls Nevertheless, we can build over 5000 packages 58
KYUA TESTS Conclusion: 2139 out of 2651 passed (81%) 59
SYSTEM ARCHITECTURE Clang Pkgsrc (libc) Pkg 1 Pkg n Users User- Land (NetBSD) VFS FS MM Rein carnat Disk Net TTY USB Servers Drivers OS (MINIX) Microkernel (this is the only part running in kernel mode) 60
MINIX 3 ON THE THREE BEAGLE BOARDS 61
YOUR ROLE MINIX 3 is an open-source project I hope some of you will join and help us Things to do Add crucial missing system calls Port more packages (Java, a browser, etc.) Write the missing drivers for Beagle series Get it running on Raspberry Pi & other platforms Port Rump Port required libraries and then port a GUI 62
MINIX 3 IN A NUTSHELL Microkernel reimplementation of NetBSD Fully open source with BSD license Highly compatible with NetBSD Supports both LLVM and gcc Uses NetBSD pkgsrc Over 5000 packages build Go get it at www.minix3.org and try it 63
POSITIONING OF MINIX Show that multiserver systems are reliable Demonstrate that drivers belong in user mode High-reliability and fault-tolerant applications $50 single-chip, small-ram laptops for 3rd world Embedded systems 64
FUTURE FEATURE: LIVE UPDATE Software is updated to: Fix bugs Improve performance Add new features Goal is to update OS to a new version w/o reboot Running processes must NOT be restarted New version of OS may have new data structures Lots of state in there: open files, timers,etc. 65
EXAMPLE OF HOW WOULD THIS WORK User A Apache running A Apache still running Kernel FreeBSD 10.1 FreeBSD 10.2 Replace the OS while user processes are running Very difficult to do with BSD, Linux, Windows, etc. 66
LIVE UPDATE IN MINIX User A Apache running A Apache still running User MM FS 6.0 MM Driver Driver FS 7.0 Kernel Microkernel Microkernel 67
HOW DO WE DO THE UPDATE? Manager tells some process (e.g. Old-FS) to get ready Old-FS finishes its work and queues new work Manager creates New-FS process with new code LLVM puts tables inside New-FS listing its data objects New-FS contacts Old-FS and asks for state it needs The state is transferred one object at a time When all state is transferred, Third-FS is created It talks to New-FS and tries to recreate Old-FS If they agree New-FS becomes FS, else revert to Old-FS Like translating English to Dutch, then Dutch to English 68
HOW THE UPDATE WORKS A Apache running FS 6.0 Old FS Microkernel 69
HOW THE UPDATE WORKS Get ready A Apache running FS 6.0 Microkernel 70
HOW THE UPDATE WORKS A Apache running FS 6.0 FS 7.0 Microkernel 71
HOW THE UPDATE WORKS A Apache running FS 6.0 I need variable x FS 7.0 G Microkernel 72
HOW THE UPDATE WORKS A Apache running FS 6.0 Here is variable x FS 7.0 Microkernel 73
HOW THE UPDATE WORKS A Apache running FS 6.0 FS 7.0 FS? Microkernel 74
HOW THE UPDATE WORKS A Apache running FS 6.0 FS 7.0 I need variable x FS? Microkernel 75
HOW THE UPDATE WORKS A Apache running FS 6.0 FS 7.0 Here is variable x FS? Microkernel 76
HOW THE UPDATE WORKS A Apache running FS 6.0 FS 7.0 FS? Are these the same? Microkernel 77
HOW THE UPDATE WORKS A Apache running FS 7.0 Microkernel 78
MUCH BETTER THAN KSPLICE KSPLICE can handle only small security patches KSPLICE patches the running process Over time, crud accumulates in the process If the update fails, there is no recovery 79
OTHER USES OF LIVE UPDATE Enhanced security: Update the OS at a high rate to foil return-to-libc attacks Stop any attack that uses knowledge of memory layout Reduce exposure to information leakage attacks Garbage collection in C (!) Only live data is copied over to the new version This can fix memory leaks (malloc but no free) 80
RESEARCH: FAULT INJECTION Inject fault? Original unmodified basic block Basic block with fault injected This structure is created automatically by the LLVM compiler 81
NEW PROGRAM STRUCTURE This can be optimized by patching the original binary to get any test without recompilation Overhead is 8% 82
MINIX 3 LOGO Why a raccoon? Small Cute Clever Agile Eats bugs More likely to visit your house than a penguin 83
WEBSITE: www.minix3.org 84
DOCUMENTATION IS IN A WIKI Wiki.minix3.org You can help document the system 85
TRAFFIC TO WWW.MINIX3.ORG Total visits to the main page since 2004: 3.1 million Actual downloads since 2007: 650,000 (from the log) 86
MINIX 3 GOOGLE NEWSGROUP 87
CONCLUSION Current OSes are bloated and unreliable MINIX 3 is an attempt at a reliable, secure OS Kernel is very small (15,000 LoC) OS runs as a collection of user processes Each driver is a separate process Each OS component has restricted privileges Faulty drivers can be replaced automatically Live update is possible (not in current release) 88
SURVEY Please download MINIX 3 from www.minix3.org Give it a try Fill out the survey on the main page We have had 650,000 downloads but we don t know who they are or what they are doing We are trying to build a community 89
THE END 90
WEBSITE: www.minix3.org 91
92
MASTERS DEGREE AT THE VU If you are interested in computer systems Look at our masters in parallel & distributed syst. Google me Look at my home page See video linked there or check out pdcs.vu.nl 93
DISK PERFORMANCE 94
THE COST OF DRIVER RECOVERY We killed the Ethernet driver every Δt sec to simulate repeated driver crashes Driver recovery takes about 360 msec 95
RESEARCH: MULTICORE CHIPS Multicore chip TCP IP Ether Kernel Core Network stack has components Chips may be heterogeneous Where to put each component? Experiments scaling frequencies Sometimes slower is faster! Sleep/wakeup is expensive 96
RESEARCH: NEW FILE SYSTEM--LORIS VFS Naming Cache Better reliabilty Better flexibility Handles heterogeneity better File rather than block oriented Uses checksums to detect corruption Logical Physical Introduces concept of a logical file (1 or more phys files spread or striped over possibly heterogeneous devices) Driver 97