Development of Type-2 Hypervisor for MIPS64 Based Systems High Performance Computing and Networking Lab Al-Khwarizmi Institute of Computer Science University of Engineering & Technology Lahore Pakistan
Agenda Introduction Scope of Project Type-II Hypervisor for MIPS64 Based Systems Design Architecture and Execution Flow Challenges Demonstration
Introduction
Virtualization Creating software representation of a device or resource E.g. server, storage device, network, OS Broadly classified as Server Virtualization Client or Desktop virtualization Network virtualization
Applications of Virtualization Cloud services Security E.g. Cloud user isolation Optimal hardware utilization E.g. Multiple services on a single machine Software development and testing E.g. Legacy code execution Dynamic load balancing in data centers E.g. Server migration
Hypervisor A piece of software/firmware Used to create Virtual Machines (VMs) Provides a virtual environment to run guest OS Type-1 (aka native, bare metal) Type-2 (aka hosted)
Scope of Project
Virtualization Solutions Virtualization software Type1 Hyper-V, Xen, Vmware ESX Type 2 Virtual PC/Server, Vmware Workstation, Virtual Box Available only for x86 A Hypervisor or Virtual Machine Monitor (VMM) for MIPS is not available
Virtualization on MIPS Relatively less explored area Recent work on type-i Longsoon MIPS hypervisor No published work on type-ii MIPS hypervisor MIPS is typically used in embedded systems E.g. Switches, routers, smart TV, set-top boxes Virtualization enabling software can increase the hardware utilization of these embedded systems
Type-II hypervisor for MIPS64 Based Systems
Important Features Full-Virtualization solution for MIPS MIPS on MIPS Virtualization enabling application Multiple MIPS64 guest OS on single Host machine running linux Multi-cores design Implementation of basic devices Developed on Cavium OCTEON family
Type-II Hypervisor Guest is a user level process of Host OS Multiple VMs could be created G. G. G. App 1 App 2... App n Guest Kernel Guest U-Boot : Hypervisor Other processes MIPS Linux MIPS Hardware
Multiple VMs Scenario VM 1 VM 2 VM n G. App 1 G. App 2... G. App n G. App 1 G. App 2... G. App n G. App 1 G. App 2... G. App n Guest Kernel Guest U-Boot Guest Kernel Guest U-Boot... Guest Kernel Guest U-Boot : : : Other processes Hypervisor Hypervisor Hypervisor MIPS Linux MIPS Hardware
Design Architecture and Execution Flow
Design Three major parts of hypervisor ISA emulation Memory management External Device implementation Combined implementation of all three results in a Virtual board
Virtual Board A Multithreaded Linux process Cores and devices are threads
Block Level ISA Emulation Basic blocks of guest instructions are extracted A stream of straight line code ending with a control shift instruction Whole block is translated to a set of safer instructions Translated block is executed on hardware Block Fetching Identify basic block Instruction Translation Replace each instruction with safer instruction(s) Block Execution Host OS executes translated code
Translation For each valid block 1. Translate each instruction of block into a set of unprivileged instructions 2. Concatenate sets of translated instructions to form the translated Block
Execution Translated block is executed directly on hardware Control may shift back to hypervisor due to following reasons: Address translation for load/store instructions TLB operations System call External interrupt End of current block Block Execution Block Fetching Block Translation
Memory Management Physical memory Virtual memory Virtual to physical memory mapping is handled by kernel Simplest virtual memory view of a system Virtual memory map Mapped kernel segment (kseg 2) Unmapped uncached kernel segment (kseg1) Unmapped cached kernel segment (kseg0) User space (kuseg)
MIPS Memory Map (64-bit View )
Guest s Memory Management Hypervisor provide 1-1 mapping of guest memory Using Mmap() system call GVA to HVA translation Identify memory segment check for GPA in TLB If found, get GPA and then Get HVA from hash-map else, generate guest exception Guest virtual address (GVA) Guest Physical Address (GPA) Host Virtual Address (HVA) Hypervisor Load/store from HVA
External Devices Multi-threaded approach for devices Implemented UART (Universal Asynchronous Receiver/Transmitter) CIU (Central Interrupt Unit) Timer Network Card
UART Used for serial communication OS transmits and receives bytes Processor communicates with console & keyboard Implementation in hypervisor Receiver thread Continuously check arrival of new data from keyboard Generate interrupt on receiving Handover the data Transmitter thread Sends data from guest to console
Central Interrupt Unit (CIU) Cavium Specific Unit Provides communication between devices and cores Routes the interrupt from devices Using summary and interrupt enable hardware registers
CIU Implementation Continuously observes devices interrupt signals Reads corresponding bits of device registers Create summary registers If any device wants to interrupt a CPU core CIU writes cause register to indicate the interrupt
On-Demand Timer Embedded in core thread Register timer on guest s demand Reads time from host and presents to guest in nanosecond resolution Much better performance
Challenges
Technical Challenges We used Cavium Octeon CN57XX as reference board CVM segment CVM specific instruction Customized instructions e.g. BBIT0, MADD, 64 bit multiplication instruction (VMULU) Partially documented Continuously pinging Cavium support staff Or doing rigorous testing on reference board
Technical Challenges Testing and verification Unexpected guest behavior e.g. halts, illegal instruction Guest instruction treated as data on host level Soln. Flush Cache after block translation Block fetching from non-contiguous pages Instruction emulation verification LL/SC, atomic load/store, madd
High Boot time Technical Challenges Hurdle in development Optimization efforts Cache implementation In-place execution Structural changes Inline functions Macros Customized data structure Improvements in Timer infrastructure
Demo
Thanks for Listening!
Ask questions to find out something about the world itself, not to find out whether or not someone knows it. John Holt