Server Virtualization Techniques Includes Slides from (Lee Badger)
genda Define Server Virtualization The Server Virtualization Sectrum Server virtualization solutions Similarities and differences OS Issues Note: Virtualization (V12N) is really a misnomer when alied to some of the HW technologies. better general term would be Workload Containment (WC) V12N is one kind of WC...HW artitioning is another...but we will use V12N generically to include both
EP Reort to Congress Server & Data Center Energy Efficiency Data center energy use more than doubled 2000-2006. The ower and cooling infrastructure accounts for 50% of data center total energy consumtion. The energy used by the nation s servers and data centers in 2006: > 61 billion kilowatt-hours (kwh) > 1.5% of total U.S. electricity consumtion! > Total electricity cost of about $4.5 billion. > Equal to 5.8 million average US Households >...and that was 2006...what do you think it is now?!
EP Reort to Congress Server & Data Center Energy Efficiency
EP Reort to Congress Server & Data Center Energy Efficiency
Motivating factors - consolidation Imroving the utilization of comuting resources > Translation: better return on the money sent One way to do this is server consolidation Consolidation requires (at least) the following > 1. Non-interference of indeendent workloads: > with security, and erformance management > 2. Resource management (to ensure service levels) > Caacity lanning is rediscovered as systems disciline! > 3. Resource accounting (to ay for shared resources) > 4. H services (since more eggs now in fewer baskets)
Goals: Server Virtualization > Run multile alication environments on the same machine at the same time without allowing them to interact (excet normal inter-machine interaction, e.g. network communication)...that is: > workload and security searation > Platform abstraction (emulation of hardware) In other words, convince the alications that they are on searate and/or different systems, even though they are sharing a system... >...actually, CPU time sharing and OS rocess scheduling does this already!
Server Virtualization Context Decoule the hard connection between this alication on this OS instance on this box essential for Cloud Comuting Virtualization has been used since the 1960s (mainframes) > Now a mainstream technology available on multile latforms Renewed emhasis due to changed economics and needs > Server srawl has gotten out of hand, energy costs have skyrocketed Different styles of V12N with different benefits and limitations Provide the Illusion of a dedicated comuter for multile OS instances: > Partitioning: hardware and/or firmware caability > Virtual machines: host OS ( hyervisor ) software (VMware, Xen,...) > Containers, zones, vservers: light-weight, single-os virtualization NOTE: Grid, J2EE, cloud comuting, service-oriented-architecture as have overlaing uroses for decouling alication from host
Other Motivators / Use Cases There are other imortant use cases for virtualization > Ugrade OS version or atch level with concurrent oeration > Migrate from one OS to another on same server > Coexist different OSes for different tyes of work > Provide searate fault, security, admin domains of same OS level > Relieve scalability constraints of a given OS via multile instances > Use legacy OS on newer systems > examles: run NT4 on current x86, Solaris 8 on SPRC T2/T3 > Develoment, sandbox, lay-en in congenial environment > Flexible, raid redeloyment of workloads to servers > For rovisioning and for motion/relocation of running workloads
Hardware Virtualization alications OS, e.g., Linux VM alications OS, e.g., Win32 VM VMM hardware... simle (simlistic) icture! VMM = Virtual Machine Monitor (Hyervisor) But imlementation is comlex. Virtual Machines (VMs) can be: susended/coied/moved/lost/recovered/hacked
V12N Terminology interretation Performing instructions written in a rogramming language (e.g., erl, ython, ruby, Java bytecodes, x86 machine code) emulation Imitating the behavior of one system (e.g., interreter) using the resources of another (e.g., les rosetta) virtualization The abstraction of comuting resources (e.g. memory, cu) virtual machine an efficient, isolated dulicate of a real machine Credit: Poek/Goldberg, Formal Requirements for Virtualizable Third Generation rchitectures
Resource Virtualization E.g, the CPU Linear address E.g, Memory a a a a directory table offset Run queue: Round robin Pre-emtive Scheduling current (10-bit offset) Pg table base (10-bit offset) (12-bit offset) Page frame Memory cell age age directory age table cr3 Credit: Intel i486 reference manual
lication Virtualization lication Software Run lication Installer Code System calls (resource request) lication V Layer ackage Run under manager Link ackage with manager Virtualization Layer simulated registry settings simulated files Running lication generate intercet generate Highly ortable. system interface simulated files simulated registry settings lication leaves no footrint on host (just user references). lication V Layer.EXE Run anywhere. VL Oerating System lication can be streamed. Isolation is voluntary. Credit: www.anandtech.com
Bytecode Interreters Java rogram Emulation OR Just-In-Time comilation n imaginary machine, excet for icojava HW Strongly tyed ~200 JVM instructions (bytecodes) Java Virtual Machine (JVM) Oerating System Hardware Safe verified on execution: valid ocodes, jum targets, tye disciline Garbage collected memory Stack-insection security Sandboxed in 2006, >4 billion JVM devices Java marketing: write once, run anywhere! Java mockery: write once, debug everywhere! (forgot who said that) Microsoft.Net Common Language Runtime (CLR) is similar but more generic.
Hardware Virtualization alications OS, e.g., Linux VM alications OS, e.g., Win32 VM VMM hardware... simle (simlistic) icture! Clearly the different VMs must be searate and secure; Why? re the VMs easier/harder to manage than real?
Hardware Virtualization (HW Server View) 1 2 dom0 OS HW OS VMM tye 1 Para-virtualization OS I/O VMM Host OS HW tye 2 OS VMM dom0 OS OS VMM HW tye 1 Full virtualization OS I/O VMM Host OS HW tye 2 OS VMM 3 Terminology OS : runs only on VMM Host OS : runs only on HW Domain : virtual machine on VMM Hyervisor : virtual machine monitor 4 x86 CPU mode ring 3 ring 2 ring 1 ring 0 Issue: Derivileging ring 3 ring 0 OS (kernel) OS (kernel) VMM ring 3 ring 1 ring 0
Hardware Virtualization Device Driver Placement dom0 tye 1 tye 2 I/O VMM OS OS ass redirect through VMM OS emulated emulated device emulated device emulated device device OS device device driver device driver device driver driver OS VMM I/O VMM Host OS OS redirect OS emulated emulated device emulated device emulated device device OS OS device VMM device driver device driver device driver driver HW HW Device Device Device Device Device Device Device Device
VMM Formal Requirements (summary of Poek and Goldberg, 1974 CCM) For machines having: 1) user/suervisor modes, 2) location-bounds register, and 3) a traing mechanism. tra memory u PC=0 (n, 4) user rogram s PC=2 (0, q-1) 0 1 2 3 4 Disatcher llocator Instruction Interreter 0 1 2 3 n n+1 n+2 n+3 n+4 q-1 Sensitive Instructions (change or deend on memory ma or mode) Privileged instructions (tra iff user mode) Efficiency: most instructions run directly. Resource Control: the VMM allocates all resources. Poek Goldberg Theorem : If then a Virtual Machine Monitor (VMM) can be built having 3 roerties: Equivalence: the user rogram mostly believes it runs on the hardware.
Making x86 Virtualizable Using Binary Translation OS kernel in ring 1 Running Basic blocks...... C B 2 Coy a newlyencountered basic block to the cache. ret C jm Translation Cache (also in memory) B call instruction instruction SGDT instruction instruction call instruction instruction instruction instruction instruction instruction instruction call OS kernel in ring 1 (if needed) VMM ring 0 1 Identify the next block by scanning instructions for a jum/call/etc (that ends a basic block). Technique used by VMware, in 1999. 3 Binary translate any rohibited instruction into a sequence that emulates it safely. 4 Run/rerun translated block at full seed.
Making x86 Virtualizable Using Extra Hardware Intel 64 Intel version of x86-64 Floating Point Data 17 rithmetic 26 Comare 14 Transcendental 8 Constants 7 Control 20 State management 2 SIMD 94 MMX 47 SSE 62 SSE2 69 SSE3 13 SSSE3 32 SSE4 54 277 General Purose Data transfer 32 rithmetic 18 Logical 4 Shift/rotate 9 Bit/byte 23 Control transfer 31 String 18 I/O 8 Enter/leave 2 Flag control 11 Segment register 5 Misc 6 167 VT-x Extensions 12 Safe mode 1 contains ~595 instructions. Hardware extensions make the instruction set virtualizable System 34 64-bit mode 10
Intel Virtual Machine Extensions (VMX) Original structure CPU mode ring 3 ring 2 ring 1 ring 0 Host OS Host OS Host OS Host OS Host OS ring 3 Host OS ring 0 Legacy software runs in the exected rings, hoefully unaware. there is no software-visible bit indicates VMX non-root oeration, Intel 64 manual. VMX non-root Derivileged (very configurable). CPU State transitions Ring 0 VMXON VMXOFF VMXLUNCH VMXRESUME VMM VMX root VMXCLL side effects Many instructions cause faultlike VM exits: interruts I/O events age table management rivileged instructions, etc. VMM handles faults VM exit rate determines erformance ddress translation is comlex
How Comlex is Virtualization? legend Source Lines Of Code 55,000,000 Debian Linux Oerating system Virtualization system 35,000,000 20,000,000 15,000,000 Windows 95 Windows 2k Windows NT Red Hat Linux 3,000,000 1,000,000 60,000 1990 Windows 3.1 Bochs Kaffe VMM code counts generated using David. Wheeler's SLOCCount tool. Windows estimate from Bruce Schneier Linux estimates from Gonzalez-Barahona et al., and David Wheeler Xen Qemu VirtualBox 2000 2008
VMM Imlementation Quality Should Not be ssumed In 2007, Tavis Ormandy subjected 6 virtualization systems to guided random testing of their invalid instruction handling and I/O emulation. Bochs QEMU VMWare Xen nonymous 1 nonymous 2 178k SLOC 373k SLOC 910k SLOC ll of the systems failed the tests, most with arbitrary execution failures. Device emulation was a articular area of vulnerability. For details, see: taviso.decsystem.org/virtsec.df Reference: n Emirical Study into the Security Exosures to Host of Hostile Virtualized Environments, by Travis Ormandy. taviso.decsystem.org/virtsec.df Code counts generated using David. Wheeler's SLOCCount tool.
Nevertheless Virtualization is now a ervasive technology Used in majority of data centers VMware on x86 has greatest market share Cometitors: Microsoft Hyer-V Xen (Oen Source, Citrix, Oracle OVM) Linux KVM but other OS & HW V12N solutions exist
Virtualization roaches Hard Partitions Hardware ssignment Hyervisors OS Virtualization Hosted Virtualization lications OS OS B VM Layer OS Features Shared OS Kernel CPU, Memory Interconnect OS OS B OS C Domain Domain B Domain C OS OS B OS C OS OS B OS C Hyervisor HW Suort? Shared Shared Shared Shared Shared Shared Shared Shared Multile OS's Single OS
Software Hyervisors lications OS Features OS Kernel CPU, Memory Interconnect Hard Partitions OS OS B OS C OS OS OS OS B OS B OS C C Domain Hyervisors Hyervisor Domain B Domain C HW Suort Shared Hardware ssignment OS V12N Shared Shared Hosted Virtualization Some cometing technologies > Tye 1 alone on the hardware > VMware ESX, KVM, Shared Shared Shared Shared Encasulation VM Layer > Xen / Citrix / Oracle OVM, > Microsoft Hyer-V > Tye 2 on an OS ( Hosted V12N ) > Virtual Box Shared > Parallels Workstation > VMware Fusion (for OS X) > Microsoft Virtual Server
User Mode Linux Overview Provides a selfcontained environment Identical as hosting Linux kernel Processes have no access to host resources that were not exlicitly rovided t r a c e VM User Process 1 Host OS Kernel VM User Process 2 OS Kernel/UML Virtual Machine System Call Intercetion
Linux KVM htt://www.linux-kvm.org Kernel-based Virtual Machine for Linux on x86 hardware containing virtualization extensions (Intel VT or MD-V) loadable kernel module Included in mainline Linux, as of 2.6.20 can run multile virtual machines running unmodified Linux or Windows images
VMotion -like Technologies VMotion-like technology lets you move live, running virtual machines from one host to another while maintaining continuous service availability. Live Migration also available on other V12N latforms Xen, SPRC T2/LDoms, IBM Power,... What are the technical challenges to imlementing this? HW? OS? lications?
Oerating System Virtualization For examle: Solaris Containers Single OS instance ( Global Container ) > earance of many OS instances... >...but not really > Minimal erformance imact Zone Zone Zone Zone Zone Zone Zone Zone Zone Zone Zone Zone Zone Zone Zone Zone Zone Zone Solaris CPU CPU CPU CPU CPU CPU CPU CPU Memory I/O I/O I/O I/O I/O I/O I/O I/O
Imact of VMs on Virtual Memory? Virtualization of virtual memory if each guest OS in every VM manages its own set of age tables? VMM searates real and hysical memory Makes real memory a searate, intermediate level between virtual memory and hysical memory OS mas virtual memory to real memory via its age tables, and VMM age tables ma real memory to hysical memory VMM maintains a shadow age table that mas directly from the guest virtual address sace to the hysical address sace of HW Rather than ay extra level of indirection on every memory access VMM must tra any attemt by guest OS to change its age table or to access the age table ointer
V12N is easy V12N Myths extra layer of training & exertise required lications run the same under V12N erformance, installation, licensing, suort can be different alications are written assuming non-virtualized OS services, PIs needed V12N requires no lanning rovision/deloy/destroy ease temts oversimlification, lack of logging V12N reduces IT infrastructure comlexity more, not less, comlex, VMs may be hard to locate without rules V12N saves money HW reduction is real, but other costs can increase (mgt SW, training) Oerating systems are dead Hyervisors are OS's...some merging of features & resonsibilities may occur;
V12N Myths V12N increases availability & reliability but H and roer failover architecture requirements and methods needed V12N enhances security VM security not yet well understood, investigation harder V12N can be used everywhere not where erformance & scalability are riorities Organizations can exloit V12N immediately not without lanning, deloyment & management training
Future of Virtualization lthough it originated decades ago, it's relatively new to the modern, multi-system data center and low- & mid-range UNIX/Linux/MS servers and workstations and to a certain extent, new to university CS curricula Many new uses...and roblems, too Much misunderstanding of imact, benefits, risks
Future of Virtualization Deskto Virtualization becoming rominent growing use of thin desktos for security, ease of deskto management back to centralized comuting model!!! why did IT move away from centralized? VM management issues tools still in develoment for rovisioning, monitoring, atching, securing, moving,... VM srawl starting to occur debugging roblems difficult non-deterministic architectures; can't relicate/isolate some roblems
Future of Virtualization Virtualized aliances reconfigured databases, web servers, a servers, thin client servers, etc encasulates OS & alication/service VM standardization OVF standard under develoment goal is to enable fully ortable VMs and their deloyment/management High vailability solutions & SLs still needed tools & services still rimitive
Future of Virtualization Continuing trend of HW-assisted V12N Intel & MD virtualization accelerators see MD s Raid Virtualization Indexing Cometition & Self-Serving Predictions Big 3 on Intel/MD: Microsoft, VMware, Xen Oerating systems are dead (VMware) rchitectural design skills needed oor level of understanding of V12N unreasonable fears & exectations