Xen: FOSS Hypervisor for the Cloud

Transcription

1 Xen: FOSS Hypervisor for the Cloud Russell Pavlicek Xen Project

2 So Who s the Fat Geek up Front?

3 About the Speaker... Linux user since 1995; Linux desktop since 1997 Linux advocate before I ever saw the software Early Linux advocate at Digital Equipment Corporation, Compaq Former FOSS columnist for Infoworld, Processor magazines Former panelist on The Linux Show webcast Wrote book, Embracing Insanity: Open Source Software Development (2000) Speaker at 50+ Open Source conferences Became Xen Project Evangelist employed by Citrix in January 2013 Formerly with Cassatt Corporation in San Jose, cloud startup ( )

4 A Word About the Xen Stack The main components: Xen Hypervisor, the central FOSS project Xen API, the Cloud enabled subproject Better known as XAPI Xen Project is a Linux Foundation Collaborative Project These are the subjects of this talk And then there s: XenServer, a popular Xen-based product Was partially closed source; recently open-sourced by Citrix

5 The Cloud Problem

6 IT: Pre-Cloud Stability is Paramount The value of IT to the corporation is consistent service availability Service capacity specified a year or more in advance What s up, stays up Change is Bad Change to status quo is disruptive and dangerous Changes are beaten into submission until they become part of the new status quo and then they are no longer changes

7 IT in the Cloud Availability of Services is Paramount The value of IT to the corporation is consistent service availability at levels matching dynamic business demand Service capacity must move with business needs What s up when depends on what s needed when Change is Good Services must change to cover the needs of the moment Lack of change = lack of value

8 Layers of the Cloud App Operating System Layer Virtualization Layer Cloud Orchestration Layer

9 Virtualization in the Cloud It must be stable It must be secure It must be configurable on a large scale The user at machine paradigm does not work If it requires a mouse, you re in trouble It must take orchestration (APIs, command line) It must be multi-tenant It must not lock you into one concept or provider of Cloud

10 Xen: Stable Solid track record Amazon s AWS cloud business uses Xen Verizon launched a new Xen-based cloud Linux Foundation Project Partners: AMD, Amazon, Cisco, Citrix, Google, Intel, NetApp, Oracle, Samsung, Verizon, and more

11 Xen: Secure SELINUX FLASK SELINUX capabilities at the VM level by the same team Disaggregation Segment device drivers into discrete VMs Architectural advantages of a Type-1 Hypervisor See my separate talk on security from CloudOpen NA 2013

12 Xen: Configurable at Scale Toolstacks give rich API and command line capabilities Not GUI-centric Empowers orchestration via scripting, power tools (Puppet, Chef, etc.), GUIs, and Cloud layers

13 Xen: Rich Toolstacks Hypervisor Xen Hypervisor Toolstack / Console Default / XL (XM) Libvirt / VIRSH XAPI / XE Increased level of functionality and integration with other components Single Host Basic Functions Single Host Additional Functionality Multiple Hosts Additional Functionality

14 Xen: Tools for Different Solutions Project Xen Hypervisor Toolstack / Console Default / XL (XM) Libvirt / VIRSH XAPI / XE Increased level of functionality and integration with other components Products Oracle VM Huawei UVP Citrix XenServer 14

15 Xen: Tools for Different Clouds Project Xen Hypervisor Toolstack / Console Default / XL (XM) Libvirt / VIRSH XAPI / XE Increased level of functionality and integration with other components Products Oracle VM Huawei UVP Citrix XenServer Used by 15

16 Focus on XAPI: The Cloud Layer Xen Hypervisor XAPI / XE Multiple Hosts Additional Functionality VM lifecycle: live snapshots, checkpoint, migration Storage XenMotion: Migrate VMs between hosts or pools without shared storage (while the VM is running) Resource pools: flexible storage and networking Event tracking: progress, notification Upgrade and patching capabilities Real-time performance monitoring and alerting Templates for Windows and Linux guests Open vswitch support built-in (default) More info: wiki.xenproject.org/wiki/xcp_release_features

17 Xen: A Multi-tenant Solution Multiple groups share common resources securely Clouds require sharing common resources Organizations often need their VMs to be visible to each other, but entirely invisible to all other VMs XAPI makes this happen Critical ability for hosting providers

18 Xen: Doesn t lock you in Xen does not force its view of the Cloud on you Xen does not force you to use a favored Cloud solution This is one of the reasons why Cloud innovation happens in the world of FOSS: It gives power to the Cloud, but allows Cloud orchestration solutions to innovate There is no attempt to bend your efforts to the will of some corporate business plan

19 XAPI : Orchestration Choices Xen Hypervisor XAPI / XE Multiple Hosts Additional Functionality

20 Xen Project Healthcheck

21 News: Xen Project in Linux Foundation See the following teams on the new XenProject.org site: Hypervisor XAPI ARM Hypervisor (for Servers as well as Mobile Devices) Mirage OS Governance : mixture between Linux Kernel and Apache Consensus decision making Sub-project life-cycle (aka incubator) PMC style structure for team leadership

22 Xen contributor community is diversifying 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% The number of significant active vendors is increasing New feature development driving new participation Citrix SUSE University GridCentric NSA Fujitsu Misc Spectralogic UPC Amazon AMD Individual Intel iweb Oracle University of British Columbia

23 More Xen Features Live Migration of VMs: XenMotion (via XAPI) High Availability: Remus Wide variety of Control Domains supported Even wider variety of Guest Domains Multiple virtualization modes improve performance

24 Hypervisor Architecture

25 Hypervisor Architectures Type 1: Bare metal Hypervisor A pure Hypervisor that runs directly on the hardware and hosts Guest OS s. VM 0 VM 1 VM n Guest OS and Apps Scheduler Hypervisor Device Drivers/Models MMU I/O Memory CPUs Host HW Provides partition isolation + reliability, higher security

26 Hypervisor Architectures Type 1: Bare metal Hypervisor A pure Hypervisor that runs directly on the hardware and hosts Guest OS s. Type 2: OS Hosted A Hypervisor that runs within a Host OS and hosts Guest OS s inside of it, using the host OS services to provide the virtual environment. VM n User-level VMM VM n VM 0 VM 1 User Apps Device Models VM 0 VM 1 Device Drivers/Models Scheduler MMU Guest OS and Apps Hypervisor Host OS Device Drivers Ring-0 VM Monitor Kernel Guest OS and Apps I/O Memory CPUs Host HW Host HW I/O Memory CPUs Provides partition isolation + reliability, higher security Low cost, no additional drivers Ease of use & installation

27 Xen: Type 1 with a Twist Type 1: Bare metal Hypervisor VM 0 VM 1 VM n Guest OS and Apps Device Drivers/Models Scheduler MMU Hypervisor I/O Memory CPUs Host HW

28 Xen: Type 1 with a Twist Type 1: Bare metal Hypervisor Xen Architecture VM n VM 1 VM n VM 0 VM 1 Scheduler Guest OS and Apps Hypervisor VM 0 Guest OS and Apps Device Drivers/Models MMU Scheduler MMU Hypervisor I/O Memory CPUs Host HW I/O Memory CPUs Host HW

29 Xen: Type 1 with a Twist Type 1: Bare metal Hypervisor Xen Architecture Control domain (dom0) VM 1 VM n Device Models VM n VM 0 VM 1 Scheduler Guest OS and Apps Hypervisor Drivers Linux & BSD VM 0 Guest OS and Apps Device Drivers/Models MMU Scheduler MMU Hypervisor I/O Memory CPUs Host HW I/O Memory CPUs Host HW

30 Xen Project and Linux Xen Hypervisor is not in the Linux kernel BUT: everything Xen and Xen Guests need to run is Xen packages in all distros (not RHEL6, but CentOS 6) Install Control Domain (Dom0) Linux distribution Install Xen package(s) or meta package Reboot Configure stuff: set up disks, peripherals, etc. More info: wiki.xenproject.org/wiki/category:host_install

31 Basic Xen Concepts Console Interface to the outside world Control domain (dom0) I/O Dom0 Kernel Scheduler MMU XSM Memory Trusted Computing Base CPUs VM 0 VM 1 VM n Guest OS and Apps Hypervisor Host HW Control Domain aka Dom0 Dom0 kernel with drivers Xen Management Toolstack Guest Domains Your apps Driver/Stub/Service Domain(s) A driver, device model or control service in a box De-privileged and isolated Lifetime: start, stop, kill 31

32 Basic Xen Concepts: Toolstack+ Control domain (dom0) I/O Toolstack Dom0 Kernel Console Scheduler MMU XSM Memory Trusted Computing Base CPUs VM 0 VM 1 VM n Guest OS and Apps Hypervisor Host HW Console Interface to the outside world Control Domain aka Dom0 Dom0 kernel with drivers Xen Management Toolstack Guest Domains Your apps Driver/Stub/Service Domain(s) A driver, device model or control service in a box De-privileged and isolated Lifetime: start, stop, kill 32

33 Basic Xen Concepts: Disaggregation Control domain (dom0) I/O Toolstack Dom0 Kernel Console Memory One or more driver, stub or service domains Scheduler MMU XSM Trusted Computing Base CPUs VM 0 VM 1 VM n Guest OS and Apps Hypervisor Host HW Console Interface to the outside world Control Domain aka Dom0 Dom0 kernel with drivers Xen Management Toolstack Guest Domains Your apps Driver/Stub/Service Domain(s) A driver, device model or control service in a box De-privileged and isolated Lifetime: start, stop, kill 33

34 Xen : Types of Virtualization

35 Xen Virtualization Vocabulary PV Paravirtualization Hypervisor provides API used by the OS of the Guest VM Guest OS needs to be modified to provide the API HVM Hardware-assisted Virtual Machine Uses CPU VM extensions to handle Guest requests No modification to Guest OS But CPU must provide VM extensions FV Full Virtualization (another name for HVM)

36 Xen Virtualization Vocabulary PVHVM PV on HVM drivers Allows H/W virtualized guests to use PV disk and I/O drivers No modifications to guest OS Better performance than straight HVM PVH PV in HVM Container (coming in 4.4) Almost fully PV Uses HW extensions to eliminate PV MMU Possibly best mode for CPUs with virtual H/W extensions

37 The Virtualization Spectrum VS VH P Virtualized (SW) Virtualized (HW) Paravirtualized Fully Virtualized (FV) VS VS VS VH FV with PV for disk & network P VS VS VH PVHVM P P VS VH PVH Xen 4.4 P P P VH Fully Paravirtualized (PV) P P P P HVM mode/domain PV mode/domain

38 The Virtualization Spectrum Optimal performance Scope for improvement Poor performance Fully Virtualized (FV) VS VS VS VH FV with PV for disk & network P VS VS VH PVHVM P P VS VH PVH Xen 4.4 P P P VH Fully Paravirtualized (PV) P P P P HVM mode/domain PV mode/domain

39 The Virtualization Spectrum Optimal performance Scope for improvement Important: Xen automatically picks the best option based on HW & OS capabilities and available drivers. As a Xen user I chose a HVM or PV domain. Poor performance Fully Virtualized (FV) VS VS VS VH FV with PV for disk & network P VS VS VH PVHVM P P VS VH PVH Xen 4.4 P P P VH Fully Paravirtualized (PV) P P P P HVM mode/domain PV mode/domain

40 Disaggregation Split Control Domain into Driver, Stub and Service Domains See: Breaking up is hard to Xen Papers See: Domain 0 Disaggregation for XCP and XenServer Used today by Qubes OS and Citrix XenClient XT Prototypes for XAPI See qubes-os.org Different windows run in different VMs

41 Benefits of Disaggregation More Security Increased serviceability and flexibility Better Robustness Better Performance Better Scalability Ability to safely restart parts of the system (e.g. just 275ms outage from failed Ethernet driver)

42 Next: XAPI Architecture Diagram Before and After Disaggregation

43 User VM User VM NF BF NF BF NB gntdev NB gntdev gntdev Dom0 Domain manager healthd xenopsd libxl Xen Network drivers networkd vswitch Dom0 NFS/ iscsi drivers storaged tapdisk blktap3 Qemu xapi Qemu Network NFS/ Local drivers iscsi storage drivers drivers qemu syslogd xapi eth eth eth eth scsi.... xapi Dom0 Xen qemu networkd vswitch storaged tapdisk blktap3 storaged tapdisk blktap3 NIC (or SR- IOV VF) NIC (or SR- IOV VF) CPU RAM RAM CPU NIC (or SR- IOV VF) NIC (or SR- IOV VF) RAID

44 User VM User VM NF BF NF BF NB gntdev NB gntdev gntdev Dom0 Domain manager healthd xenopsd libxl Network driver domain networkd vswitch NFS/ iscsi driver domain storaged tapdisk blktap3 Qemu domain qemu xapi domain xapi D o m 0... Logging domain syslogd Network driver domain networkd vswitch NFS/ iscsi driver domain storaged tapdisk blktap3 Local storage driver domain storaged tapdisk blktap3 dbus over v4v dbus over v4v eth eth eth eth scsi Xen Xen NIC (or SR- IOV VF) NIC (or SR- IOV VF) CPU RAM RAM CPU NIC (or SR- IOV VF) NIC (or SR- IOV VF) RAID

45 Xen Security Advantages Even without Advanced Security Features Well-defined trusted computing base (much smaller than on type-2 HV) Minimal services in hypervisor layer Xen Security Modules (or XSM) and FLASK XSM is Xen equivalent of LSM (Linux Security Modules) FLASK is Xen equivalent of SELinux Developed, maintained and contributed to Xen by NSA Compatible with SELinux (tools, architecture) XSM object classes maps onto Xen features More info: a-brief-tutorial-on-xens-advanced-security-features 45

46 Xen Security Modules (XSM) -- FLASK What does FLASK provide? Granular security Can a guest domain talk with other guest domains? Can a guest domain only communicate with the Control Domain? Can a Guest domain have memory which cannot be read by the Control Domain? What type of device model is used in this domain? The ability to define multiple security roles on the domain level User types can be defined and assign roles Policy constraint logic More info: 46

47 User VM User VM NF BF NF BF NB gntdev NB gntdev gntdev Dom0 Domain manager healthd xenopsd libxl Network driver domain networkd vswitch NFS/ iscsi driver domain storaged tapdisk blktap3 Qemu domain qemu xapi domain xapi D o m 0... Logging domain syslogd Network driver domain networkd vswitch NFS/ iscsi driver domain storaged tapdisk blktap3 Local storage driver domain storaged tapdisk blktap3 dbus over v4v dbus over v4v eth eth eth eth scsi FLASK policy restricting access Xen Xen NIC (or SR- IOV VF) NIC (or SR- IOV VF) CPU RAM RAM CPU NIC (or SR- IOV VF) NIC (or SR- IOV VF) RAID

48 ARM Hypervisor

49 Xen 4.3 for ARM Servers Fully functional for ARM v7 & v8 ARM v7: Versatile Express, Arndale & Samsung Chromebook ARM v8: Fast Model

50 Xen + ARM = a perfect Match ARM SOC ARM Architecture Features for Virtualization User mode : EL0 Device Tree describes I/O Hypercall Interface HVC Kernel mode : EL1 GT GIC v2 2 stage MMU Hypervisor mode : EL2

51 Xen + ARM = a perfect Match ARM SOC ARM Architecture Features for Virtualization EL0 Device Tree describes I/O HVC EL1 GT GIC v2 2 stage MMU Xen Hypervisor EL2

52 Xen + ARM = a perfect Match ARM SOC ARM Architecture Features for Virtualization Device Tree describes I/O Any Xen Guest VM (including Dom0) User Space Kernel HVC EL0 EL1 GT GIC v2 2 stage MMU Xen Hypervisor EL2

53 Xen + ARM = a perfect Match ARM SOC ARM Architecture Features for Virtualization I/O Dom0 only Device Tree describes PV back PV front Any Xen Guest VM (including Dom0) User Space Kernel EL0 EL1 HVC GT GIC v2 2 stage MMU Xen Hypervisor EL2

54 One mode to rule them all Optimal performance Scope for improvement x86: PVHVM P P VS VH x86: PVH P P P VH HVM mode/domain PV mode/domain ARM v7 & v8 P VH VH VH

55 Code Size of x86 and ARM Hypervisors X86 Hypervisor 100K -120K LOC Any x86 CPU ARM Hypervisor for mobile Devices ARM Hypervisor for Servers 60K LOC ARM v5 v7 (no virtual extensions) (extra code for RT) 17K LOC ARM v7+ (w/ virtual extensions)

56 Mirage OS

57 Library Operating Systems Application stacks only running on Xen APIs Works on any Xen based cloud or hosting service Examples ErlangOnXen.org : Erlang HalVM : Haskell Mirage OS : Ocaml Benefits: Small footprint Low startup latency Extremely fast migration of VMs Control domain (dom0) PV Back Ends HW Drivers Dom0 Kernel Guest VM n Apps Library OS embedded in Language run-time Xen Host HW

58 Mirage OS Recently added to Xen Project incubator In beta stage: first release on its way Clean-slate protocols implementations, e.g. TCP/IP, DNS, SSH, Openflow (switch/controller), HTTP, XMPP,... New applications using next generation XAPI (disaggregated XAPI architecture) More info: mirage-extreme-specialisation-of-virtual-appliances

59 What s Next?

60 New in Xen 4.3 (July 2013) Xen ARM for Servers Extend scope of Xen Security Modules qxl Spice support for 3d acceleration Updated and improved libvirt drivers for Xen Lots of other stuff: scalability, performance, better NUMA support, Next Release Info:

61 What s next (and already happening) Establish a shared test infrastructure Most major contributors are duplicating effort Mirage OS provides interesting opportunities Usability and better distribution integration Example: Xen + XAPI in CentOS 6.4 More focus on downstreams Examples: CloudStack and Xen Orchestra Disaggregation in Xen toolstacks (e.g XAPI) Better Libvirt and virt-manager integration Embed Xen more into the Linux ecosystem and provide benefits for the wider Linux community

62 Getting Started with Xen Project Document Days (monthly) Test Days (prior to release) Mailing Lists, IRC, Newsletter XenProject.org (sign up, it s free!) Xen Hackathon, Next one expected Spring 2014 Xen Developer Summit, 10/25-26, Edinburgh Xen User Summit, last month, New Orleans

63 News: blog.xenproject.org Web: XenProject.org Help for IRC, Lists, Stackoverflow-like Q&A Wiki: wiki.xenproject.org Presentations: slideshare.net/xen_com_mgr Videos: vimeo.com/channels/xen Thank Slides available under CC-BY-SA 3.0 From