MIT IAP Course Lecture #1: Virtualization 101

Transcription

1 MIT IAP Course Lecture #1: Virtualization 101 Carl Waldspurger (SB SM 89 PhD 95) VMware R&D January 16, 2007

2 What is Virtualization? vir tu al (adj): existing in essence or effect, though not in actual fact Virtual systems Abstract physical components using logical objects Dynamically bind logical objects to physical configurations Examples Network Virtual LAN (VLAN), Virtual Private Network (VPN) Storage Storage Area Network (SAN), LUN Computer Virtual Machine (VM), simulator 2

3 Overview Virtual Machines Virtualization Approaches Processor Virtualization Additional Topics 3

4 Starting Point: A Physical Machine Physical Hardware Processors, memory, chipset, I/O bus and devices, etc. Physical resources often underutilized Software Tightly coupled to hardware Single active OS image OS controls hardware 4

5 What is a Virtual Machine? Hardware-Level Abstraction Virtual hardware: processors, memory, chipset, I/O devices, etc. Encapsulates all OS and application state Virtualization Software Extra level of indirection decouples hardware and OS Multiplexes physical hardware across multiple guest VMs Strong isolation between VMs Manages physical resources, improves utilization 5

6 VM Isolation Secure Multiplexing Run multiple VMs on single physical host Processor hardware isolates VMs, e.g. MMU Strong Guarantees Software bugs, crashes, viruses within one VM cannot affect other VMs Performance Isolation Partition system resources Example: VMware controls for reservation, limit, shares 6

7 VM Encapsulation Entire VM is a File OS, applications, data Memory and device state Snapshots and Clones Capture VM state on the fly and restore to point-in-time Rapid system provisioning, backup, remote mirroring Easy Content Distribution Pre-configured apps, demos Virtual appliances 7

8 VM Compatibility Hardware-Independent Physical hardware hidden by virtualization layer Standard virtual hardware exposed to VM Create Once, Run Anywhere No configuration issues Migrate VMs between hosts Legacy VMs Run ancient OS on new platform E.g. DOS VM drives virtual IDE and vlance devices, mapped to modern SAN and GigE hardware 8

9 Common Virtualization Uses Today Test and Development Rapidly provision test and development servers; store libraries of pre-configured test machines Server Consolidation and Containment Eliminate server sprawl by deploying systems into virtual machines that can run safely and move transparently across shared hardware Business Continuity Reduce cost and complexity by encapsulating entire systems into single files that can be replicated and restored onto any target server Enterprise Desktop Secure unmanaged PCs without compromising end-user autonomy by layering a security policy in software around desktop virtual machines 9

10 Overview Virtual Machines Virtualization Approaches Virtual machine monitors (VMMs) Virtualization platform types Alternative system virtualizations Processor Virtualization Additional Topics 10

11 What is a Virtual Machine Monitor? An Old Concept Classic definition from Popek & Goldberg 74 IBM mainframes since 60s VMM Characteristics Fidelity Performance Isolation / Safety 11

12 VMM Technology So this is just like Java, right? No, a Java VM is very different from the physical machine that runs it A hardware-level VM reflects underlying processor architecture Like a simulator or emulator that can run old Nintendo games? No, they emulate the behavior of different hardware architectures Simulators generally have very high overhead A hardware-level VM utilizes the underlying physical processor directly 12

13 VMMs Past From IBM VM/370 product announcement, ca An Old Idea Hardware-level VMs since 60s IBM S/360, IBM VM/370 mainframe systems Timeshare multiple single-user OS instances on expensive hardware Classical VMM Run VM directly on hardware Trap and emulate model for privileged instructions Vendors had vertical control over proprietary hardware, operating systems, VMM 13

14 VMMs Present VMware Fusion for Mac OS X running WinXP, 2006 Renewed Interest Academic research since 90s VMs for commodity systems Server consolidation VMM for x86 Industry-standard hardware, from laptops to datacenter Run unmodified commodity guest operating systems Significant challenges, e.g. non-virtualizable instructions Pioneered by VMware in 98 14

15 VMM Platform Types Hosted Architecture Install as application on existing x86 host OS, e.g. Windows, Linux, OS X Small context-switching driver Leverage host I/O stack and resource management Examples: VMware Player/Workstation/Server, Microsoft Virtual PC/Server, Parallels Desktop Bare-Metal Architecture Hypervisor installs directly on hardware Acknowledged as preferred architecture for high-end servers Examples: VMware ESX Server, Xen, Microsoft Viridian (2008) 15

16 System Virtualization Alternatives Virtual machines abstracted using a layer at different places Language Level OS Level Hardware Level 16

17 System Virtualization Taxonomy System Virtualization Hardware Level High-Level Language Bare-Metal/ Hypervisor HP Integrity VM IBM zseries z/vm VMware ESX Server Xen Hosted Para-virtualization Virtual Iron VMware VMI Xen Microsoft Virtual Server Microsoft Virtual PC Parallels Desktop VMware Player VMware Workstation VMware Server OS Level FreeBSD Jail HP Secure Resource Partitions Sun Solaris Zones SWsoft Virtuozzo User-Mode Linux Java Microsoft.NET / Mono Smalltalk Emulators Bochs Microsoft VPC for Mac QEMU Virtutech Simics 17

18 Overview Virtual Machines Virtualization Approaches Processor Virtualization Classical techniques Software x86 VMM Hardware-assisted x86 VMM Para-virtualization Additional Topics 18

19 Classical Instruction Virtualization Trap and Emulate Run guest operating system deprivileged All privileged instructions trap into VMM VMM emulates instructions against virtual state e.g. disable virtual interrupts, not physical interrupts Resume direct execution from next guest instruction Implementation Technique This is just one technique Popek and Goldberg criteria permit others 19

20 Classical Memory Virtualization guest VMM VPN PPN MPN shadow page table hardware TLB Traditional VMM Approach Extra Level of Indirection Virtual Physical Guest maps VPN to PPN using primary page tables Physical Machine VMM maps PPN to MPN Shadow Page Table Composite of two mappings For ordinary memory references Hardware maps VPN to MPN Cached by physical TLB 20

21 Memory Traces Shadow Page Table Derived from primary page table in guest VMM must keep primary and shadow coherent Trace = Coherency Mechanism Write-protect primary page table Trap guest writes to primary Update or invalidate corresponding shadow Transparent to guest 21

22 Classical VMM Performance Native Speed Except for Traps No overhead in direct execution Overhead = trap frequency average trap cost Trap Sources Most frequent: Guest page table traces Privileged instructions Memory-mapped device traces 22

23 x86 Virtualization Challenges Not Classically Virtualizable x86 ISA includes instructions that read or modify privileged state But which don t trap in unprivileged mode Example: POPF instruction Pop top-of-stack into EFLAGS register EFLAGS.IF bit privileged (interrupt enable flag) POPF silently ignores attempts to alter EFLAGS.IF in unprivileged mode! So no trap to return control to VMM Deprivileging not possible with x86! 23

24 How to Virtualize x86? Interpretation Problem too inefficient x86 decoding slow Code Patching Problem not transparent Guest can inspect its own code Binary Translation (BT) Approach pioneered by VMware Run any unmodified x86 OS in VM Extend x86 Architecture 24

25 Software VMM: Binary Translation Direct execute unprivileged guest application code Will run at full speed until it traps, we get an interrupt, etc. Binary translate all guest kernel code, run it unprivileged Since x86 has non-virtualizable instructions, proactively transfer control to the VMM (no need for traps) Safe instructions are emitted without change For unsafe instructions, emit a controlled emulation sequence VMM translation cache for good performance 25

26 VMware Translator Properties Binary input is x86 hex, not source Dynamic interleave translation and execution On Demand translate only what about to execute (lazy) System Level makes no assumptions about guest code Subsetting full x86 to safe subset Adaptive adjust translations based on guest behavior 26

27 BT Mechanics Input: BB 55 ff 33 c Each Translator Invocation Consume a basic block (BB) Produce a compiled code fragment (CCF) Store CCF in Translation Cache translator Future reuse Capture working set of guest kernel Amortize translation costs Not patching in place Output: CCF 55 ff 33 c

28 Example: IDENT Translation 80304a69 push %ebp 80403a6a push (%ebx) 80403a6c mov (%ebx), ffffffff 80403a72 mov %edx, %esp 80403a74 mov %esp, 81c(%ebx) 80403a7a push %edx 80403a7b mov %ebp, %eax 80403a7d call 80460ba4 BB 25555b0 push %ebp 25555b1 push (%ebx) 25555b3 mov (%ebx), ffffffff 25555b9 mov %edx, %esp 25555bb mov %esp, 81c(%ebx) 25555c1 push %edx 25555c2 mov %ebp, %eax 25555c4 push 80403a c9 int 3a 25555cb data: 80460ba4 CCF 25555c4: push return address 25555c9: invoke translator on callee 28

29 Adaptive BT!*! Translation Cache Translated Code Is Fast Mostly IDENT translations Runs at speed Except Writes to Traced Memory Page fault (shown as!*!) Decode and interpret instruction Fire trace callbacks Resume execution Can take 1000 s of cycles Invoke Translator 29

30 Adaptive BT: Fast Trace Handling JMP TRACE Detect and Track Trace Faults Splice in TRACE Translation Execute memory access in software Avoid page fault No re-decoding Faster resumption Faster Traces 10x performance improvement Adapts to runtime behavior Invoke Translator 30

31 Software VMM Evaluation Benefits Adaptation Fast traces Fast I/O emulation Flexibility Costs Running translator Path lengthening System call slowdown Complexity 31

32 Hardware-Assisted VMM Recent x86 Extension : Software-only VMMs using binary translation 2005: Intel and AMD start extending x86 to support virtualization First-Generation Hardware Enables classical trap-and-emulate VMMs Intel VT, aka Vanderpool Technology AMD SVM, aka Pacifica Performance VT/SVM help avoid BT, but not MMU ops (actually slower!) Main problem is efficient virtualization of MMU and I/O, Not executing the virtual instruction stream 32

33 VT/SVM Architecture CPL 3 CPL 2 CPL 1 CPL 0 Host CPL 3 CPL 2 CPL 1 CPL 0 Guest Diagram Y-axis: old school x86 privilege (CPL) X-axis: virtualization privilege Guest Mode Runs unmodified OS Sensitive operations exit (trap out) to host mode VMCB Virtual Machine Control Block VMM-controlled, hardware-walked Buffers simple exits 33

34 Hardware-Assisted VMM Hardware-Assisted Direct Exec CPL 0-3 Guest mode Fault, Trace, Interrupt, I/O... Resume Guest VMM CPL 0-3 Host mode 34

35 Hardware-Assisted VMM Evaluation Benefits Simplicity (no BT) Fast system calls No translator overheads Costs Exits: 1000 s of cycles for traces and I/O No adaptation or software flexibility Stateless model Future Hardware support for fast MMU virtualization Intel EPT, AMD NPT 35

36 What is Paravirtualization? Full Virtualization No modifications to guest OS Excellent compatibility, good performance, but complex Paravirtualization Exports Simpler Architecture Term coined by Denali project in 01, popularized by Xen Modify guest OS to be aware of virtualization layer Remove non-virtualizable parts of architecture Avoid rediscovery of knowledge in hypervisor Excellent performance and simple, but poor compatibility Ongoing Linux Standards Work Paravirt Ops interface between guest and hypervisor Small team from VMware, Xen, IBM LTC, etc. 36

37 Paravirtualization: Conceptual Diagram Guest OS Guest OS System call interface Hypercalls (GOOD) Hypervisor Hardware Hypervisor Hardware NOT GOOD! Full Virtualization Paravirtualization 37

38 VMware Vision: Transparent Paravirtualization Same OS binary Dom0 VMI Linux DomU Xeno Linux VMI Linux Windows Solaris VMI Linux Xen 3.0.x VMware ESX Native Native Native 38

39 Further Reading VMware Publications A Comparison of Software and Hardware Techniques for x86 Virtualization (ASPLOS 06) Fast Transparent Migration for Virtual Machines (USENIX 05) Memory Resource Management in VMware ESX Server (OSDI 02) Virtualizing I/O Devices on VMware Workstation s Hosted VMM (USENIX 01) Additional Academic Publications Xen and the Art of Virtualization (SOSP 03) Disco: Running Commodity Operating Systems on Scalable Multiprocessors (SOSP 97) Many more 39

40 Additional Topics I/O Virtualization Memory Management 40

41 I/O Virtualization Stack Guest OS Device Driver Device Emulation I/O Stack Device Driver Guest Device Driver Virtual Device Model existing device, e.g. e1000 Model an idealized device, e.g. vmxnet Virtualization Layer Emulates the virtual device Remaps guest and real I/O addresses Multiplexes and drives physical device Provides additional features, e.g. transparent NIC teaming Real Device Physical hardware, e.g. bcm5700 Likely to be different than virtual device 41

42 I/O Virtualization Implementations Guest OS Hosted or Split Emulated I/O Hypervisor Direct Guest OS Passthrough I/O Guest OS Device Driver Device Driver Device Driver Device Emulation Host OS/Dom0/ Parent Domain Device Emulation Device Emulation I/O Stack Device Driver I/O Stack Device Driver Device Manager VMware Workstation, VMware Server, VMware ESX Server (for slow devices), Xen, Microsoft Viridian, Virtual Server VMware ESX Server (storage and network) A Future Option Many Challenges 42

43 Passthrough I/O Virtualization Guest OS Device Driver I/O MMU Device Manager VF VF VF I/O Device Virtualization Layer Guest OS Device Driver PF PF = Physical Function, VF = Virtual Function Guest OS Device Driver High Performance Guest drives device directly Minimizes CPU utilization Enabled by HW Assists I/O-MMU for DMA isolation e.g. Intel VT-d, AMD IOMMU Partitionable I/O device e.g. PCI-SIG IOV spec Challenges Hardware independence Migration, suspend/resume Memory overcommitment 43

44 Additional Topics I/O Virtualization Memory Management 44

45 Memory Management Desirable capabilities Efficient memory overcommitment Accurate resource controls Exploit sharing opportunities Challenges Allocations should reflect both importance and working set Best data to guide decisions known only to guest OS Guest and meta-level policies may clash 45

46 VMware Memory Management Reclamation mechanisms Ballooning guest driver allocates pinned PPNs, hypervisor deallocates backing MPNs Swapping hypervisor transparently pages out PPNs, paged in on demand Page sharing hypervisor identifies identical PPNs based on content, maps to same MPN copy-on-write Allocation policies Proportional sharing revoke memory from VM with minimum shares-per-page ratio Idle memory tax charge VM more for idle pages than for active pages to prevent unproductive hoarding 46

47 Ballooning inflate balloon (+ pressure) Guest OS balloon may page out to virtual disk Guest OS balloon guest OS manages memory implicit cooperation deflate balloon ( pressure) Guest OS may page in from virtual disk 47

48 Page Sharing Motivation Multiple VMs running same OS, apps Collapse redundant copies of code, data, zeros Transparent page sharing Map multiple PPNs to single MPN copy-on-write Pioneered by Disco [Bugnion 97], but required guest OS hooks Content-based sharing General-purpose, no guest OS changes Background activity saves memory over time 48

49 Page Sharing: Scan Candidate PPN VM 1 VM 2 VM 3 hash page contents 2bd806af Machine Memory hint frame Hash: VM: PPN: MPN: 06af 3 43f8 123b hash table 49

50 Page Sharing: Successful Match VM 1 VM 2 VM 3 Machine Memory shared frame Hash: Refs: MPN: 06af 2 123b hash table 50