Operating Systems Part M15 Virtual Memory, Shared Libraries, Microkernels, Sandboxing and Virtualization. Florina Ciorba 24 November 2015

Transcription

1 Operating Systems Part M15 Virtual Memory, Shared Libraries, Microkernels, Sandboxing and Virtualization Florina Ciorba 24 November 2015

2 Today s Overview (Main) Memory management Shared libraries Compiling, Linking, Loading Microkernel (continuation) Virtualization (intro) Operating Systems Concepts 2

3 New Chapter: (Main) Memory Management Idealized memory management for Unix (example Linux) Kernel Virtual Memory Stack Memory-mapped region Memory-mapped region Memory-mapped region Run-time data Uninitialised data Initialised data Program text Memory invisible to user mode code the brk pointer Linux: Linear address space Regions: code (text) initialized variables (data) uninitialized variables (bss) Heap (dynamic allocation) Libraries Stack (FILO) Kernel Forbidden region Operating Systems Concepts 3

4 Main Memory Management: Problem Statement Limited physical memory, often too small Representation of logical (memory) address space onto the physical memory When is one memory address represented as a memory cell? Compilation, linking Program loading Program execution Virtual memory: Binding is achieved at run time which has to be translated by the hardware (for every access) Operating Systems Concepts 4

5 Repetition: Swapping Swapping = Storing of an (entire) process on the disk Unix: Swap partition, Windows: swap file operating system 1 swap out process P 1 2 swap in process P 2 user space backing store main memory Operating Systems Concepts 5

6 Virtual Memory Virtual memory = separation of logical memory from the physical memory possibilities (RAM, hard disk) Operating Systems Concepts 6

7 Virtual Memory (Continued) Execute processes whose memory footprint is larger than the main memory (RAM) Memory parts of a process can be found In physical memory (RAM) On the hard disk (swap space) Transfer between hard disk and main memory Swapping vs (demand) paging Which program/ page is in main/secondary memory? When to transfer which program/page? Operating Systems Concepts 7

8 Memory Management: Paging In a multiprogramming system Multiple processes live at the same time in memory (co-exist) Upon execution of a program the created memory zones are not contiguous: How can they be merged? Partition physical memory in frames and logical memory in pages (size 0.5 until 4 KB) Map logical page areas to physical frames Address translation Operating Systems Concepts 8

9 Memory Management: Paging (Continued) Paging = Memory management for non contiguous (virtual) memory logical address physical address Mapping of pages (memory pages ) to frames (memory frames ) Page Table CPU p d f d p physical memory Address translation via hardware: memory management unit (MMU) f page table Operating Systems Concepts 9

10 Memory Management: Page Table (Example) frame number page 0 0 page page 0 page page 3 page table 3 page 2 logical memory 4 page page 3 physical memory Operating Systems Concepts 10

11 Memory Management: Page Table Implementation Page table itself needs 1 frame, remains in main memory (RAM) Is read by the MMU (memory management unit) at each memory access Content of a page table entry Attribute (mapped or not, read-only, swapped) frame number (or block number in case it is deposited on hard disk) Now dual memory access! Dual cache directory (reverse table of the cache) Accessible with physical memory addresses Contains a backpointer to the cache line Cache required: TLB (translation look-aside buffer) Operating Systems Concepts 11

12 Valid/Accessed/Dirty Bits Page table for book keeping Until now in the page table: frame number Additional attributes present-bit (i.e., mapped to main memory) Otherwise: invalid, or on the hard disk #define _PAGE_PRESENT 0x001 #define _PAGE_PSE 0x080 #define _PAGE_RW 0x002 #define _PAGE_GLOBAL 0x100 #define _PAGE_USER 0x004 #define _PAGE_UNUSED1 0x200 #define _PAGE_PWT 0x008 #define _PAGE_UNUSED2 0x400 #define _PAGE_PCD 0x010 #define _PAGE_UNUSED3 0x800 #define _PAGE_ACCESSED 0x020 #define _PAGE_FILE 0x040 #define _PAGE_DIRTY 0x040 Operating Systems Concepts 12

13 Memory Management: Multilevel Page Tables The page tables are too large in the case of large virtual memory 2 GB with 4 KB pages: 0.5 M entries Each process needs its own page table Therefore, multilevel page tables Allow holes similar to sparse files (see next slide) Similar to index file allocation scheme (last lecture) Operating Systems Concepts 13

14 Memory Management: Multilevel Page Tables (Cont.) 0. outer-page table page of page table page table memory Operating Systems Concepts 14

15 Memory Management: Multilevel Page Tables (Cont.) Example 4 KB pages, 32-bit architecture (32-bit addresses) Distribution Page number: 20 bits Page offset: 12 bits Page number in two parts each 10 bits: addr (32 bits) = p1 (10 bits) p2 (10 bits) offs (12 bits) p1: Index in outer page table p2: Index in inner page referenced via p1 Operating Systems Concepts 15

16 Memory Management: Tricks with Page Tables Shared pages (shared code, shared memory) Available only one time in physical memory Appears multifold in the process address space Memory protection Each process has its own virtual memory Access outside of allocated zones program termination Dynamic distribution Stack and heap: brk limit Swapping Store individual pages on the hard disk, while info in the page table ( frame in main physical memory) Operating Systems Concepts 16

17 Memory Management: Segments / Pages Segment concept Various zones (code, data), therefore segments Address = segment descriptor + offset CPU keeps track of multiple segments: segment table (Physical) location of the stack segments, code segments, OS segments, etc. Implicit referencing: stack segment, code segment Intel x86 architecture Since x286: segments Since x386: paged segments Linux: linear address space (only one segment) Operating Systems Concepts 17

18 Page Fault Virtual memory is larger than physical memory Hardware (MMU) identifies access on not mapped pages Page fault trap is raised Operating system can investigate basic causes Invalid address area (zone)? This can be important (multiple zones) Otherwise demand paging : page will be fetched from hard disk Operating Systems Concepts 18

19 Process of a Page Fault hard disk user program main memory Operating Systems Concepts 19

20 Page Fault Handling: Challenges Requires restart of the command/instruction Simplest case: Page fault due to an operand First fetch the page, then execute the instruction (e.g., previous slide) Harder case: Page fault due to a result Operation needs to be re-executed Problems with page effects (e.g., Y++) Further larger page effects can be envisioned Copy operations of the entire memory area (e.g., perform partial copy backwards?) Operating Systems Concepts 20

21 Paged Kernel Memory? Can parts of the OS be paged? Yes! For data (e.g., data buffers) In principle also possible for code Performance, etc. Operating Systems Concepts 21

22 Copy-On-Write (COW) Process start under Unix fork() Creates a child process out of the parent process Only the memory map is copied, not the content Reference to shared information Page tables of both processes point to the same memory In addition The RW flag will be removed (this means read only page ) At the next page fault The child process prepares own copy The parent process re-sets the RW flag Operating Systems Concepts 22

23 On Demand Paging Pages are loaded only when they are required. Process start Initialize memory map Fill data area Load only first (code) pages Hand over control to process Further (code) pages will be fetched on demand Operating Systems Concepts 23

24 Quote Introduced at the Beginning of the Lecture The Linux memory manager implements demand paging with a copy-on-write strategy relying on the 386 s paging support. A process acquires its page tables from its parent (during a fork()) with the entries marked as read-only or swapped. Then, if the process tries to write to that memory space, and the page is a copy-on-write page, it is copied, and the page is marked read-write. An exec() results in the reading in of a page or so from the executable. The process then faults in any other pages it needs. Linux Kernel Hacker s Guide 0.5 Operating Systems Concepts 24

25 On Demand Paging Page Replacement Multiprogramming: Competition for free frames Multiple programs are active Which one receives how many (physical) frames? Which pages will be fetched from hard disk? Activity and age of a page Which pages were sampled (ACCESSED Bit)? Determine the least recently used (LRU) page Various algorithms to approximate LRU (full LRU would be too expensive) Operating Systems Concepts 25

26 Thrashing Paging works due to locality One process usually works with only a few pages These pages move slowly When a process has too few pages in main memory (while the others are on the hard disk) High page-fault rate low CPU utilization Thrashing: insufficient locality of reference A process is busy with paging (in and out) Physical memory insufficient Disk access times overly long Also affects other processes (less CPU and I/O bandwidth) Computer performance degrades or collapses Operating Systems Concepts 26

27 Thrashing (Continued) CPU utilization thrashing degree of multiprogramming Thrashing behavior can be provoked by a simple program Define a large array (> physical main memory) Perform randomized access to individual elements of the array Operating Systems Concepts 27

28 Compiling, Linking, Loading Compiling: produces object code Certain addresses are still to be determined (external functions) Certain addresses still need to be processed (relocation) Linking: produces a binary Solves unresolved external references Classic: static linking Loading: loading of the (binary) program into the main memory (RAM) Memory allocation (eventually only few pages usually code, if paging on demand) Relocation Operating Systems Concepts 28

29 Shared Libraries, Overview of Variants This is code reuse/sharing: Similar to file sharing (last lecture) Advantages of shared libraries: shared objects (*.so, *.dylib) Same (physical) memory for multiple processes Simple system-wide upgrade (change one file, no recompilation or relinking) Additional definitions after compiling, linking, and loading Dynamic loading Dynamic linking Operating Systems Concepts 29

30 Optimization: Dynamic Loading Binaries can become very large due to (shared) libraries: lazy loading Better memory usage Do not load unused subroutines Implementation Caller checks first whether routine is already loaded If not, the run time loader is called Run time loader updates internal address table The call to the routine can be made No OS support required: Linker (which produces the binaries) can implement this itself. Operating Systems Concepts 30

31 Optimization: Dynamic Linking Multiple programs use the same subroutine, which are only stored once in main memory: lazy linking OS needs to provide support (separate address spaces) Keeps track of all loaded libraries in a list Takes precautions at link time Library call goes to a substitute routine ( stub ) Stub must first locate or even load the subroutine Afterwards, replace the stub Followed by direct jump into the routine Operating Systems Concepts 31

32 Dynamic Linking Implementation Linux OS needs to know the location of libraries: ldconfig and /etc/ld.so.conf (Unix) dyldinfo (Mac OS X) See also ld.so (dynamic linker/loader) To analyze the resolute process: ldd or otool -L Windows: DLL (dynamically linked library) Operating Systems Concepts 32

33 End of Memory Management: Memory Mapping Further tricks using memory mapping Shared memory Fast inter-process communication (IPC) No copying as with pipes Memory mapped files All data represented in memory Paging takes care that only required blocks are in fact loaded Operating Systems Concepts 33

34 Microkernel Recall? Monolithic operating system: A large program that comprises all kernel functions Microkernel Small software with overview over the hardware This allows the OS parts to work together Advantage (and prerequisite) Smaller, modular, and more portable kernel Disadvantage Loss of speed due to the modules borders Operating Systems Concepts 34

35 Microkernel (Continuation) OSF/1 OS/2 4.3 BSD HPUX database system tasks and threads IPC virtual memory scheduling Mach Concurrently run multiple larger OS kernels! (OS personality) Operating Systems Concepts 35

36 The MACH Microkernel (1983) Goals Support for SMP (parallel Unix) Modularization of Unix Unix variants can simultaneously drive Communication between components or ports and message exchange System call = Send message to the kernel Exception = Send message to a task Tasks can execute on multiple CPUs, eventually coupled through a network Operating Systems Concepts 36

37 MACH (Continued) Unix main module Runs as module (process) outside of the microkernel! In addition (as parallel user space processes) File system server module Access to the Unix file system Memory manager (!) Responsible for paging and swapping Unix server Responsible for processes, permissions, libraries, etc. User processes Operating Systems Concepts 37

38 MACH: System Call for Unix Trampoline technique: Library can decide on its own how it wants to work through an SVC. Operating Systems Concepts 38

39 Distributed OS vs Network OS Adoption: Network (this means no SMP here) Distributed OS (Mach, Chorus) The OS functionality is in itself distributed Illusion of a single computer (single login) Network OS Each computer has a copy of the OS Login required on every machine Common file system, password file, etc. Another name: Cluster Operating Systems Concepts 39

40 Virtualization Definition Virtualization = Framework, in which the computing resources are either combined and/or distributed, to give a uniform picture of one or more environments Examples of virtualization Time sharing Virtual memory (Partial) Simulation or emulation of a CPU General concept Map M real resources to N virtual resources, N usually larger than M Operating Systems Concepts 40

41 Applications of Virtualization Administration of server pools, application pools, Cloud! Sandboxing (isolation, controlled execution environment) Development (virtual HW) Replacement of failing HW Debugging Software emigration (mobility) SW Test, SW Evaluation, SW Quality Control Operating Systems Concepts 41

42 What Have We Learned Today? Shared libraries Compiling, Linking, Loading Microkernel (continuation) Virtualization (intro) Operating Systems Concepts 42