Solaris Virtualization and the Xen Hypervisor Frank Hofmann

Solaris Virtualization and the Xen Hypervisor Frank Hofmann Solaris Released Products Engineering Sun Microsystems UK

All things in the world come from being. And being comes from non-being. Lao Tzu

Overview Virtualization techniques by Sun Microsystems > Hardware partitions/domains > Virtual Servers > System Emulation > Hypervisors Logical partitions/domains A little bit on sun4v and quite a lot on i86xen Solaris Xen dom0 / domu interfaces Peeking at Xen in action

Telescope view at Virtualization Virtual Environment A Virtual Environment B... Virtual Environment X Upcalls The black box of The System

Virtualization by Service Java Virtual Machine write once run everywhere Application servers automatic service discovery Webserver farms load balancing NFS server failover high availability Very high level approach at virtualization beyond scope today...

Virtualization by Interface Filesystem Hierarchy: Virtual Servers / Zones System calls: dosemu, Wine, BrandZ Instruction set: JVM, MacOS Rosetta, FX!32 Hardware: Simics, VMware, Parallels, QEmu,... Paravirtualization: Hypervisors Most real products are hybrids.

@Sun: Hardware Domains / Partitioning Requires Support on-board for things like: > Complete CPU/Memory/IO board hotplugging > Full hardware redundancy First available on SPARC hardware by CS6400, E10k Hardware fault isolation high cost in hardware and adminstration Number of Hardware domains (severely) limited Not Virtualization, just Consolidation (more than one system in a box)

@Sun: Zones (Virtual Servers) Multiple administrative domains Number of Zones not limited to hardware capabilities Zone creation/bringup/deletion is very cheap Software/Application fault isolation Single OS instance, single hardware Per-Zone Resource Management: > CPU share reservation > Memory reservation (not yet shipping in Solaris 10) > Storage reservation > Virtual Network interfaces

@Sun: Branded Zones Multiple administrative domains System call interface interposition Currently provides Linux Zones Possible future extensions: > Wine Zone > *BSD Zone > old Solaris Zone > Add instruction emulator and create SPARC Zone?

@Sun: System Emulation SPARC CPUs instruction emulation > full software SPARC possible by specification > For example: UltraSPARC-T1 has no VIS unit: http://cvs.opensolaris.org/source/xref/on/usr/src/uts/sun4/os/visinstr.c > No SPARC CPU implements full sparcv9 instruction set VMware ESX Server certified on Sun Galaxy

@Sun: System Emulation SPARC CPUs instruction emulation > full software SPARC possible by specification > For example: UltraSPARC-T1 has no VIS unit: http://cvs.opensolaris.org/source/xref/on/usr/src/uts/sun4/os/visinstr.c > No SPARC CPU implements full sparcv9 instruction set VMware ESX Server certified on Sun Galaxy BAD ATTEMPT AT MARKETING BY AN ENGINEER...

Paravirtualization and Hypervisors App A... App X App... App Zone Y Zone X Logical Domain A Operating System Logical Domain B Operating System System Calls... Logical Domain X Operating System Hypercalls Virtual Machine Monitor / Hypervisor Direct Hardware Access The black box of The Hardware

@Sun: sun4v Logical Domains UltraSPARC-T1 (Niagara) Firmware has Hypervisor Solaris on sun4v architecture: LDoms Virtual Network / Disk devices on T1000/T2000 Sourcecode: http://cvs.opensolaris.org/source/xref/on/usr/src/uts/sun4v/ Interface Specifications: http://www.opensparc.net/publications/presentations/your-os-on-the-t1-hyperv http://opensparc-t1.sunsource.net/

@Sun: Solaris/Xen, domu and dom0 Why do we do it? > Because it's cool! > Because it's cool! > Because it's really cool! > For boring talk, ask your Sun Sales Representative What do we provide? > Solaris as Hypervisor Solaris dom0 in Xen terms > Solaris as Guest Solaris domu (new architecture: i86xen) Project Codename: No idea... really...

Xen Components Overview Virtual Network Driver Virtual Disk Driver Xen Guest (domu) HAT Layer Code Xen Event Channels Xen Hypercalls Virtual Network Backend Virtual Disk Backend Xen Host (Dom0) Direct Hardware Access The black box of The Hardware

Solaris/Xen: Dom0 Hypervisor code Solaris/Xen dom0 code: > communication between domu/dom0 via: > interrupt handler (hypercalls) > shared memory ringbuffers (xenstore) > Xen interdomain communication abstraction layer: Event Channels > key functions/sourcefiles in server side (dom0): usr/src/uts/i86xen/os/evtchn.c, xen_callback_handler() usr/src/uts/common/xen/io/xenbus*.c usr/src/uts/common/xen/io/xendev.c usr/src/uts/common/xen/os/xvdi.c > Caveat: very complex code structure. Hypervisor host/guest code not in separate sourcefiles!

Solaris/Xen: virtual network Solaris/Xen virtual network driver: > Both frontend and backend are network drivers > Frontend domu: usr/src/uts/common/xen/io/xnf.c > Backend dom0: usr/src/uts/common/xen/io/xnb.c

Solaris/Xen: virtual disks Solaris/Xen virtual disk driver: > Virtual Block Device, VBD: > Frontend domu: usr/src/uts/common/xen/io/xvbd.c > Backend dom0: usr/src/uts/common/xen/io/xvbdb.c

Solaris/Xen: DomU hypervisor interface Solaris/Xen hypercall consumers, domu code: > I've run out of time... > This is a workshop so why not look while playing? > We've got a whole day to find out > Use the source, luke! > Self-study starts with: find. -type f -exec grep hypercall '{}' /dev/null \; <131 files> find. -type f -exec grep HYPERVISOR '{}' /dev/null \; <459 files>

References Xen Project: http://xen.sf.net/ http://www.cl.cam.ac.uk/research/srg/netos/xen/ http://www.xensource.com/ OpenSolaris Xen Community: http://www.opensolaris.org/os/community/xen/ http://www.opensolaris.org/os/community/xen/how-to-8-15-06/ OpenSPARC, sun4v hypervisor: http://opensparc-t1.sunsource.net/ Intel VT and AMD Pacifica (lots more...): http://www.intel.com/technology/itj/2006/v10i3/3-xen/ 5-performance-tuning-vt-x.htm http://developer.amd.com/articles.jsp?id=14&num=1

When I let go of what I am, I become what I might be. Lao Tzu

Solaris Virtualization and the Xen Hhypervisor Frank Hofmann Frank.Hofmann@sun.com