Virtualization in Linux. DCLUG talk Przemek Klosowski October 2011

Transcription

1 Virtualization in Linux DCLUG talk Przemek Klosowski October 2011

2 Outline History and background Technology pieces : virtualization and emulation Linux implementation history Xen KVM Virtualization basic facilities Techniques and tricks Related technologies (storage, networking, I/O) Management of large setups

3 Virtualization: short history IBM : System/360+CP/CMS in the 1960s Each user gets its own 'computer' Motivation: OS development on expensive computer Widely successful with end-users PC/Intel: yeah, you get a real COMPUTER Unfortunate result: forgot lessons of virtualization VMware: Stanford OS research + Intel arch work Showed demand; caused re-invention of x86 architectural virtualization features (positive feedback between performance and popularity)

4 Technology Virtual resource: either IS THERE but PRETENDS IT'S NOT or ISN'T THERE but PRETENDS IT IS Computational state: memory, CPU registers Handled by MMU hardware and OS software I/O state (disks, network, other i/o devices) Handled by trapping privileged instructions and hypervisor software events: interrupts, faults

5 Detection and emulation Detection: detect and take over when the thread accesses a virtual resource step by step emulation (slow)... or paravirtualization... or hardware support Emulation: look what the thread was trying to do and handle it Requires parsing and emulation of native binary Needed even with hardware virtualization (just traps)

6 Emulation and paravirtualization Emulation: read and interpret every machine instruction Useful in other contexts: existed for emulation of x86 code on other platforms (PowerPC, MIPS, Sparc, ARM; DEC Alpha and Itanic even had hardware support) Bochs (Kevin Lawton) and QEMU (Fabrice Bellard) Paravirtualization: modify program/os, substituting parts needing virtualization with direct emulation calls Assumes friendly,cooperative relationship with clients

7 Virtualization hardware support Original x86 architecture not virtualized Requires para-virtualization or emulation; messy. Intel VMX and AMD SVM architecture extensions CPU flags from /proc/cpuinfo 64-bit (lm flag) MMU virtualization: nested page tables (Intel EPT and AMD RVI) I/O virtualization: PCI device pass-through (AMD IOMMU/Intel VT-d) single-root PCIe device multiplexing (SR-IOV)

8 Problems Emulation Slow Inefficient (consider loops) Can use tricks: Paravirtualization Lazy emulation a la Borland floating point Thaw-melt emulation like Java VM Requires patching (source-code type relationship) Unreliable/unsecure But can be faster than the alternatives Also can speed up hardware virtualization

9 Linux implementations Vmware, out of Berkeley/Stanford, 1998; bought by EMC in 2004 Xen by Xensource/U.Cambridge, ~2003, bought by Citrix in 2007 Hypervisor (Dom0) running DomU guests KVM by Qumranet/Israel, ~2005, bought by RedHat in 2008 KVM is just a kernel module; no need for hypervisor Qmranet CTO Moshe Bar developed OpenMOSIX, was a co-founder of Xensource, senior editor at Byte, Tel Aviv U. prof and IDF officer

10 VMware Virtualized set of hardware: video, net, disk, USB, sound, serial/parallel, etc Doesn't require VM hardware support Well-done commercial quality software Curb appeal of good management tools free/beer components but escalating pricing for entire package

11 Xen Hypervisor-based: Ring0 Stripped down OS that just provides services Boots, runs first, special, privileged guest: Dom0 Management of guests, communications Guests run in Ring1 (or 3): DomU Paravirtualized, using hypervisor calls Or virtualized on VT-x/AMD-V, via common abstraction layer QEMU provides virtual devices: PIIX3 IDE, Cirrus VGA, RTL8139 and NE2000 network, PAE, ACPI

12 KVM KVM is a kernel module, device driver for /dev/kvm System task uses it to Set guest memory space Request virtual/simulated I/O (disk, net, USB, sound) Map video buffer to host's video device QEMU provides emulation, as always Cirrus VGA, i440fx PCI, PIIX3 ISA, Sound Blaster 16, Am7990/E1000/NE2000/RTL8139 Comes with network-aware SPICE protocol

13 Kernel pieces: /dev/kvm System ioctls: query and set global attributes which affect the whole kvm subsystem create virtual machines VM ioctls: query and set attributes that affect an entire virtual machine, e.g. memory layout create virtual cpus (vcpus) vcpu ioctls: query and set attributes that control the operation of a single virtual cpu like, actually RUN something See kernel/documentation/kvm/api.txt

14 I/O, networking and other system pieces

15 QEMU Started as user-mode emulator (programs) Can also do computer/system emulations (real/extended mode, I/O) Dynamic code evaluator/interpreter Supports multiple architectures: SPARC,MIPS,ARM,PowerPC,SHx, MicroBlaze, hardware-assisted Loongson-3 Uses intermediate TCG code format Defines its own disk format QCOW/QCOW2: dynamic/demand provisioning

16 Basic VM operations Simplest boot: running memtest Qemu-kvm kernel /boot/memtest Running Universal Boot CD with FreeDOS live images: Qemu-kvm cdrom /tmp/ubcd.iso Booting Fedora live from ISO image, using a QCOW2 disk image: Qemu-kvm cdrom /tmp/fedoralive.iso hda testvm.img boot d

17 libvirt A GPL toolkit to interact with the virtualization capabilities of recent versions of Linux (and other OSes) long term stable C API set of bindings for common languages CIM provider for the DMTF virtualization schema QMF agent for the AMQP/QPid messaging system

18 libvirt supports KVM/QEMU, Xen, LXC and OpenVZ Linux container system, and User Mode Linux paravirtualized kernel VirtualBox, VMware ESX, GSX, Workstation and Player hypervisors, Microsoft Hyper-V hypervisor, Virtual networks using bridging, NAT, VEPA and VN-LINK storage on IDE/SCSI/USB disks, FibreChannel, LVM, iscsi, NFS and filesystems

19 virsh Commandline interface to libvirt supports Xen, QEmu, KVM, LXC, OpenVZ, VirtualBox, OpenNebula, and Vmware ESX. virsh version virsh list virsh create virsh migrate

20 virt-manager

21 Disk images QEMU has its own VM thin provisioned/ondemand format, QCOW/QCOW2 Creation qemu-img create -f qcow2 test.img 4G iscsi for remote disks Maintenance kpartx -av /dev/vdisks/vg1 Shared/network storage ISCSI initiator (client) and target (server)

22 Networking Bridging Ipchains FORWARD NAT Complex setups: multiple networks Storage network (iscsi, NFS) Fast, jumbo frames, etc Data network (to the outside) Management network (migration, config) Maybe others: cluster network

23 Migration Will be moving/cloning vmorig to vmnew on another machine virsh dumpxml vmorig > vmnew.xml Edit the template: Change name to vmnew Change UUID and MAC Make sure both computers share the VM storage virsh define vmnew.xml virsh start vmnew virsh migrate vmnew live qemu+ssh://station2/system

24 Management of large setups What is a 'Large' setup: dozens or hundreds of virtual machines Associated storage and networking Migration Snapshots and backups of VMs Redundancy/high availability VM pools and deduplication of storage Virtualization management VMware Virtual Center Redhat Virtualization Manager