Hybrid Virtualization The Next Generation of XenLinux Jun Nakajima Principal Engineer Intel Open Source Technology Center
Legal Disclaimer INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. INTEL PRODUCTS ARE NOT INTENDED FOR USE IN MEDICAL, LIFE SAVING, OR LIFE SUSTAINING APPLICATIONS. Intel may make changes to specifications and product descriptions at any time, without notice. All products, dates, and figures specified are preliminary based on current expectations, and are subject to change without notice. Intel, processors, chipsets, and desktop boards may contain design defects or errors known as errata, which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. *Other names and brands may be claimed as the property of others. Copyright 2007 Intel Corporation. Throughout this presentation: VT-x refers to Intel VT for IA-32 and Intel 64 VT-i refers to the Intel VT for IA-64, and VT-d refers to Intel VT for Directed I/O 2
Xen Architecture with VT-x 3P 1/3P X86-64 XenLinux Control Panel (xm/xend) Native Device Drivers Domain 0 Device Models Backend Virtual driver Domain N X86-64 XenLinux Front end Virtual Drivers Guest VM (VMX) (32-bit) Unmodified OS FE Virtual Drivers Guest BIOS Virtual Platform Guest VM (VMX) (64-bit) Unmodified OS FE Virtual Drivers Guest BIOS Virtual Platform 3D 0D Callback / Hypercall VMExit VMExit 0P Event channel Control Interface Scheduler Event Channel Hypercalls Processor Memory I/O: PIT, APIC, PIC, IOAPIC, RTC Device models Physical Host Hardware Xen Hypervisor 3
Para-Virtualization Software interface to virtual machines that is similar but not identical to that of the underlying hardware: CPU, MMU, I/O, time, interrupt handling, etc. Requires modifications to the guests para_virt osp is already in Linux (i386) VMM specific code Applicable even to hardware-assisted virtualization as well Use hypercalls E.g. FE virtual drivers Software-only para-virtualization Obviates hardware-assisted virtualization XenLinux today is based on software-only para-virtualization 4
Disadvantages of Software-Only Para-Virtualization Modified CPU Behaviors Hard for the kernel developers to fully understand virtual CPU Ring compression, no privileged instructions, no page-level protection (kernel in ring 3), interrupt masking/unmasking, More complex than the native CPU! The functionality/definition of virtual CPU can be different on different VMMs (VMware vs. Xen, for example) Or ever changing, not well-defined Overheads Fast system calls are no longer fast Exceptions/interrupts are always intercepted by the hypervisor Inefficient TLB usage No global pages for the kernel 5
XenLinux Maintenance Issues Significant modifications to Linux Rebase costs Paravirt infrastructure in Linux helps, but paravirt_ops are too large (>70 ops) Always behind the mainstream Device drivers and hardware support can be missing Does not run on the native Installation Different kernel binaries The standard Linux needs to run as XenLinux Dom0 and DomU Dom0 will need Xen-specific drivers 6
Hardware-Assisted Full Virtualization No modifications to the guest operating systems Improve the robustness, and possibly performance compared with softwarebased techniques Performance is acceptable as long as we don t run enterprise/server applications 7
Disadvantages of Hardware- Assisted Full Virtualization Requires a system with the feature But available on almost all new x86-64 based systems High CPU utilization, especially with I/O and MMU Affects scalability Server consolidation might not be so effective Software-only para-virtualization is often faster with I/O or memory intensive workloads (or both) 8
Solution Hybrid Virtualization Use hardware-assisted virtualization Consistent and well-defined CPU behavior Regain the native CPU optimization features lost in software-only para-virtualization Benefit from future silicon enhancements for hardwareassisted virtualization More features, lower VM entry/exits costs Use para-virtualization on the focused areas Reduce virtualization overheads Simplify the implementation VMM-Agnostic Single para-virtualization code in Linux for various VMMs Common binary as the native 9
VMM-Agnostic Para-Virtualization Detected by CPUID (e.g. leaf 0x4000_00xx) on x86/x86-64 Never detected by the native Pseudo H/W features MMU (e.g. direct paging mode, page sizes) I/O Interrupt controllers Time/idle SMP (IPI) Allow the same guest Linux binary to use the single para-virtualization code across VMMs Otherwise, detect a VMM, then use the the VMM-specific para-virt code 10
Mem Virtualization: General Principles VM 0 VM 1 CR3 PD PT CR3 PD PT Guest OS PT Guest OS PT VMM Memory Virtualization Host Hardware TLB Memory Guest OS expects to control address translation Allocates memory, page tables, manages TLB consistency, etc. But, VMM must have ultimate control over phys mem Must map guest-physical address space to host-physical space Security: Memory Isolation is implemented by VMM 11
IA-32 Address Translation TLB D R/W U/S CR3 PD PDE... PT PTE... F F Paging-related Control Registers CR0 PE, PG, WP VPN PFN Access Hardware sets A / D Bits PT PTE... F F CR4 CR2 PAE, PSE Faulting Address PFN D A U/S R/W P IA-32 defines a hierarchical page-table structure Defines linear-to-physical address translation After page-table walk, page-table Entries (PTEs) are cached in a hardware TLB IA-32 address translation configured via control registers (CR3, etc.) Invalidation of PTEs signaled by OS via INVLPG instruction 12
Virtualizing Page Tables: Some Options Option 1: Protect access to guest-os page tables (PTs) Use paging protections or binary translation to detect changes Upon write access, substitute remapped phys address in PTE Also need VM exit on page-table reads (to report original PTE value to guest OS) Option 2: Make a shadow copy of page tables Guest OS freely changes its page tables VM exit occurs whenever CR3 changes VMM copies contents of guest page tables to active page tables Option 3: Direct paging mode Expose guest to host (and vise versa) address mapping (modifications to guests) The page tables are write-protected, and modifications are done by hypercalls No need to construct shadow page tables 13
More Efficient than Software-only Para-Virtualization The kernel regains the native CPU features lost in softwareonly para-virtualization Fast system calls global pages Paging-based protection (U/S), etc. Privileged instructions GDT, IDT, LDT, TSS, cli/sti, etc. Standard exceptions/interrupts Efficient handling in hardware-assisted virtualization Example: Error Code Filtering #PF in user mode is raised directly to the guest kernel without causing a VM exit CR3 target caching I/O bitmap, MSR bitmap VPID/ASID 14
Para-virtualization vs. HW features Use HW virtualization features if more efficient Example: VMM disables direct paging mode if EPT/NPT is available (and faster) CPUID.EAX.0x4000_0010 returns 0 Falls back to the native paging mode 15
Initialization in Hybrid- Virtualization Linux Boots from the real mode like the native Same code as the native Detect para-virtualization support Use CPUID (leaf 0x4000_00xx) to detect the features Switch to the direct paging mode if detected MMU is switched from the shadow page table mode to the direct paging mode Modified to use 4KB pages for the kernel mapping Use the paravirt_ops for MMU (paravirt_mmu_ops) Enable other para-virtualization (pseudo HW) features as detected 16
usec usec Preliminary Performance* (lmbench micro benchmarks on x86-64) Smaller is better 7000 12 6000 10 5000 8 para domu 4000 para domu 6 hybrid 3000 hybrid KVM 4 KVM 2000 2 1000 0 0 fork proc exec proc sh proc null call null I/O stat open/close sig inst sig hndl *Para domu:= Xen software-only domu (XenLinux), hybrid:= hybrid virtualization Linux, KVM := unmodified Linux on KVM 17
Current Status Initial prototype on Xen Para_virt ops based on Xen-API Some benchmarks from Lmbench are close to native (no way for SW-based x86-64 XenLinux) Kernel build performance is same as domu Re-measuring performance using the latest processors Slightly better than domu. Prototype on KVM Started running Uses para_virt ops for x86-64 (MMU only at this point) 18
Summary Hybrid-virtualization reduces virtualization overheads using para-virtualization for hardware-assisted virtualization Same or better performance than software-only para-virutalization Minimize modifications to Linux Much fewer para_virt ops Common binary Linux the hypervisor == Linux the guest (KVM) Para-virtualization activated when running as a guest VMM-agnostic Single para-virtualization code in Linux for various VMMs VMMs have different implementations inside XenLinux will be the standard Linux running in hardware-assisted virtualization 19
3D 1/3D Domain 0 (VMX) X86-64 Linux Control Panel (xm/xend) Native Device Drivers Device Models Xen Architecture with Hybrid- Virtualization Backend Virtual driver Domain N (VMX) FE Virtual Drivers X86-64 Linux Guest BIOS Virtual Platform Guest VM (VMX) (32-bit) Unmodified OS FE Virtual Drivers Guest BIOS Virtual Platform Guest VM (VMX) (64-bit) Unmodified OS FE Virtual Drivers Guest BIOS Virtual Platform 3D 0D VMExit VMExit VMExit VMExit 0P Event channel Control Interface Scheduler Event Channel Hypercalls Processor Memory I/O: PIT, APIC, PIC, IOAPIC, RTC Device models Physical Host Hardware Xen Hypervisor 20
21