The Freescale Embedded Hypervisor

Transcription

1 November, 2010 The Freescale Embedded Hypervisor Jacques Landry Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

2 Agenda AMP Considerations Technical Overview of the Freescale Embedded Hypervisor epapr & Device Trees Overview Porting an OS to the Freescale Embedded Hypervisor Hypervisor Boot Flow Summary Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 2

3 AMP Considerations Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

4 Partitioning Multicore Systems Multicore System Hardware CPU CPU CPU CPU Memory Shared Cache Interrupt Controller I/O I/O I/O I/O I/O Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

5 Symmetric Multiprocessing (SMP) Same OS image runs on all CPUs Linux Multicore System Hardware CPU CPU CPU CPU Memory Shared Cache Interrupt Controller I/O I/O I/O I/O I/O Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

6 Asymmetric Multiprocessing (AMP) Different OS images runs on all CPU Linux RTOS Legacy OS Multicore System Hardware CPU CPU CPU CPU Memory Memory Memory Shared Cache Interrupt Controller I/O I/O I/O I/O I/O Memory Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

7 Unsupervised AMP (Asymmetric Multiprocessing) Requires partitioning hardware resources: Private resources: CPUs, memory, I/O devices Shared resources: memory, devices Doing this cooperatively (all operating systems well-behaved) presents challenges Linux RTOS Legacy OS Multicore System Hardware CPU CPU CPU CPU Memory Memory Memory Shared Cache Interrupt Controller I/O I/O I/O I/O I/O Memory Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

8 Supervised AMP Hypervisor Software Analogous to role of an operating system kernel in managing user processes More privileged than operating systems Enforces system security Manages globally shared resources Virtualizes some resources e.g. interrupt controller, UART Hypervisor/Supervisor partition partition partition Linux RTOS Legacy OS Multicore System Hardware CPU CPU CPU CPU Memory Memory Memory Shared Cache Interrupt Controller I/O I/O I/O I/O I/O Memory Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

9 Use Cases Virtualization enables running multiple OSes on a system at the same time Why do this? Oversubscription or underutilized resources Run multiple OSes on a single underutilized CPU Consolidation Multiple operating systems/partitions on a single multicore chip Multiple homogeneous operating systems in an AMP configuration on multiple cores Divided workload (e.g. control plane, data plane) Multiple operating systems, possibly heterogeneous, need to work securely and seamlessly together. Isolation mechanisms are needed for safety, robustness. Efficient inter-partition communication mechanisms are needed for cooperation. Isolate untrusted software/sandboxes Migration Migrate functionality from legacy RTOS to another OS (e.g. Linux). Security Secure partition for sensitive security tasks (e.g. access rights control, rule definitions, key storage/management) Guest OS CPU Guest OS hypervisor Guest OS CPU Memory Memory Guest OS hypervisor CPU Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

10 Unsupervised AMP Considerations Security all OSes must be trusted and well behaved because any OS can map any physical address How will global resources be initialized, shared, and/or used? How are global events handled? MPIC who initializes? PAMU who will initialize and set up PAACT table? How will PAACT configuration be specified/described? Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

11 CPC (Platform Cache) Partitioning who initializes? Corenet Coherency Domains How are they set up and initialized Unsupervised AMP Considerations continued Scarce Resources P4080 has only 2 duarts, GPIO pins Datapath Initialization how are Fman, Bman, Qman, PME initialized? OS assumptions about physical address 0x0? Debugging how to debug multiple OSes at the same time? What communications channel? JTAG? Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

12 error management Unsupervised AMP Considerations continued who will set up configurable error parameters? (e.g. single bit ECC error threshold) in DDR, CPC, etc where is the platform error interrupt (interrupt 0) routed to? How is it handled? Error conditions: global DDR, CPC, CCM, internal SRAM How are device errors handled? PCIE, Qman, Bman, Fman who handles PAMU access violations (interrupt 8)? If one OS crashes, how will recovery occur? /How will other OSes know? Boot what is the boot sequence for starting all OSes? Do they all start at the same time? Is there a 'primary' OS that takes care of global initialization? How will the sequencing work? How are secondary CPUs booted? Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

13 Technical Overview of the Freescale Embedded Hypervisor Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

14 Freescale Embedded Hypervisor partition partition partition Linux RTOS Legacy OS Hypervisor Multicore System Hardware CPU CPU CPU CPU Memory Memory Memory Shared Cache Interrupt Controller I/O I/O I/O I/O I/O Memory Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

15 Freescale s Embedded Hypervisor A small hypervisor for embedded systems based on Power Architecture technology (architecture version 2.06) Initial version focuses on static partitioning CPUs, memory and I/O devices can be divided into logical partitions Partitions are isolated from one another Configuration is fixed until a reconfigure and system reboot Not addressing problem of multiple operating systems on 1 CPU Uses the Embedded Hypervisor feature in the QorIQ/e500mc which makes virtualization efficient Uses a combination of full-virtualization and para-virtualization which provides good performance and minimal changes to guest operating systems Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

16 Hypervisor Contrasts Freescale Hypervisor Implementation Traditional Hypervisor Implementation Guest OS Guest OS Guest OS Guest OS CPU CPU CPU Requirement: supervised AMP -- isolation, performance Implications: No more than one OS per core, OS has direct control of high-speed peripherals Requirement: high level of virtualization-- solves problem of under-utilized CPUs, plus isolation Implications: more than one OS per core, complexity, performance implications QorIQ P4080 hypervisor hardware assists in meeting both requirement sets Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

17 Operating System sees a virtual core plus hypervisor services Hypervisor Features Virtual CPU (like e500mc minus hypervisor features) guest operating system Services via hypercall Emulation (privileged instructions) hypercalls device tree Debug stub interface for debugging guest operating systems virtual CPU (e500vcpu) Hypervisor services PIC Partition Mgmt Doorbells GPIO IOMMU boot services (epapr) Byte Channels direct I/O Debug console debug stub mux UART UART device tree system hardware Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

18 MMU CPU to Memory Access Control MMU is controlled by hypervisor and restricts all CPU accesses to physical address space Linux CPU CPU Memory Access Denied RTOS CPU CPU Memory Access OK I/O I/O I/O Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

19 IOMMU Device to Memory Access Control IOMMU enforces I/Oto- memory accesses Linux RTOS A key component in a securely partitioned system CPU Memory CPU Access Denied CPU CPU Memory Access OK I/O IOMMU I/O I/O Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

20 I/O device is assigned and dedicated to partition Direct I/O PCI Serial RapidIO DMA USB SD/MMC Guest Hypervisor driver physical hardware Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

21 Direct I/O thru Portal P4080 data path portal is assigned and dedicated to partition Network interfaces Security Pattern matcher Guest Guest A driver Guest B driver Hypervisor portal portal physical hardware Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

22 Virtual I/O Hypervisor Interrupt controller I 2 C GPIO Byte-channels Guest Hypervisor driver hypercall or emulation driver physical hardware Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

23 Boot Services epapr guest operating system Emulation (privileged instructions) hypercalls device tree virtual CPU (e500vcpu) Hypervisor services PIC Partition Mgmt Doorbells GPIO IOMMU boot services (epapr) Byte Channels direct I/O Debug console debug stub mux UART UART device tree system hardware Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

24 Boot Services epapr epapr (Embedded Power Architecture Platform Requirements) Defines boot program to client program interface Boot program: firmware, hypervisor Client program: bootloader, hypervisor, OS Device tree Data structure that represents a partition s hardware and virtual resources CPUs, memory, I/O devices, hypervisor-provided resources Address passed to boot CPU Multi-CPU boot architecture Single boot cpu Mechanisms to start secondary CPUs Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

25 Hypervisor Device Trees HW, HV, Guest Guest #1 Guest #2 Guest Dev tree Guest Dev tree dynamically created and loaded into guest memory HW Dev tree hypervisor HV config tree Loaded into hypervisor memory by u- boot. Hardware dev tree /chosen node points to HV config tree u-boot Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

26 Guest Device Tree A data structure used for representing a partition s physical and virtual devices root cpus cpu0 example See application note AN Understanding Device Tree Files in Multicore Hypervisor/LWE Implementations cpu1 memory virtual interrupt controller shared memory AN3649 contains specific details on both hypervisor and partition device trees hypervisor byte-channel doorbell Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

27 Core Enumeration From the perspective of a partition, the CPUs are contiguous. However, the actual physical CPUs being utilized may be noncontiguous. For example, the master Hypervisor tree may make physical CPUs 1, 3, and 5 available to a partition. However, from the perspective of the partition the available CPUs are viewed as CPUs 0, 1, and 2. Additionally, no CPUs other than those explicitly made available to a partition are visible from within the partition. Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 27

28 Virtual CPU guest operating system Emulation (privileged instructions) hypercalls device tree virtual CPU (e500vcpu) Hypervisor services PIC Partition Mgmt Doorbells GPIO IOMMU boot services (epapr) Byte Channels direct I/O Debug console debug stub mux UART UART device tree system hardware Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

29 Virtual CPU The behavior of CPU facilities (instructions/registers/interrupts) as seen by an OS-- an e500mc minus the hypervisor extensions Full virtualization is used i.e. OS is not hypervisor aware with respect to CPU behavior there are some exceptions to this general rule User and kernel mode privileged instructions and registers behave normally Hypervisor privileged instructions and register accesses trap to hypervisor and are emulated by the hypervisor User mode Kernel mode Hypervisor mode lwz r3,(r4) addi r3,r3,1 sc system call mfspr r3,spr_dec privilege trap Hypervisor Decrementer emulation Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

30 Hypervisor Services hcalls guest operating system Emulation (privileged instructions) hypercalls device tree virtual CPU (e500vcpu) Hypervisor services MPIC Partition Mgmt Doorbells Power Error boot services (epapr) Byte Channels direct I/O Debug console debug stub mux UART UART device tree system hardware Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

31 Hypervisor Services hcalls Interrupt controller (MPIC) Byte-channels character I/O stream Inter-partition signaling doorbell Partition management Start/stop/image-loading Partition management interrupts Reset Power management change clock frequency, power states Error Management Future CPU hot plug/unplug GPIO supports partitioning of GPIO pins IOMMU supports create/destroy mappings Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

32 VMPIC API Hypercall FH_VMPIC_SET_INT_CONFIG FH_VMPIC_GET_INT_CONFIG FH_VMPIC_SET_MASK FH_VMPIC_GET_MASK FH_VMPIC_GET_ACTIVITY FH_VMPIC_IACK FH_VMPIC_EOI Configures the specified interrupt Description Returns the configuration of the specified interrupt Sets the mask for the specified interrupt source Returns the mask for the specified interrupt source Returns a value indicating the activity status of an interrupt source, regardless of whether an interrupt has been requested or is in service. Acknowledges an interrupt and retrieves the interrupt number. Signals the end of processing for the highest-priority interrupt currently in service. Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

33 Byte-Channels Byte-channel a hypercall based character I/O channel Flexible endpoint configuration A physical UART on the QorIQ P4080 Another byte-channel endpoint A byte-channel to UART multiplexer A hypervisor debug stub The hypervisor console partition Debug console Host UART telnet partition byte-channel Hypervisor debug stub Byte-channel mux server telnet partition gdb (host) partition Bytechannel mux UART RS232 Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

34 Debug Stubs Hypervisor provides an internal API that allows debug stubs to be created and built into the hypervisor. Currently mutually exclusive from guest debug mode, where a guest owns the CPU debug resources (debug interrupt and registers) Two stubs supported today: gdb TRK (Code Warrior) Each stub controls 1 CPU Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

35 Debug Stub Event Flow partition partition OS OS Hypervisor stub stub stub MUX Host System Hardware CPU Memory CPU CPU Memory GDB GDB MPIC UART Memory MUX server GDB remote serial protocol Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

36 Inter-partition Signaling Mechanism by which operating systems can signal each other partition partition partition partition A one-way signal with no payload which results in an external interrupt in the destination partition send endpoint receive endpoint Hypervisor receive endpoints receive endpoint send endpoint One-to-many, manyto-one supported Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

37 Capabilities Copy data to/from another partition s memory (e.g. loading OS images) Starting other partitions Rebooting other guests Notifications guest watchdog fires, guest requests reboot, error conditions Partition Management partition partition partition running Linux running stopped RTOS stopped running Legacy OS Hypervisor Multicore System Hardware CPU CPU CPU CPU Memory Memory Memory Shared Cache Interrupt Controller I/O I/O I/O I/O I/O Memory Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

38 Privilege Levels Guest State (GS) MSR bit Under Hypervisor Bare Metal partition partition User MSR[PR=1][GS=1] User MSR[PR=1][GS=0] Kernel/Supervisor MSR[PR=0][GS=1] OS Kernel/Supervisor MSR[PR=0][GS=0] OS Hypervisor MSR[PR=0][GS=0] CPU CPU Memory Memory I/O I/O Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 38

39 Standards power.org epapr 1.0 complete in 8/2008 Resource discovery (device tree) Multi-CPU boot power.org Embedded Virtualization Committee Virtual CPU standard the behavior of instructions and registers under a hypervisor Working on RFC to the Power ISA targeting 2.07 Paravirtualization & standard hcalls Device tree related hypervisor node Shared page mechanisms Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

40 epapr & Device Trees Overview Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

41 Boot Program / Client program operating system Boot firmware device tree guest operating system Hypervisor device tree device tree Boot firmware Boot Program Firmware Second stage bootloader Hypervisor Client Program Second stage bootloader Hypevisor Operating system Other bare metal application Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

42 epapr 1.0 Power.org released the Power Architecture Platform Requirements (PAPR) specification in August for desktop/server platforms epapr 1.0 for embedded systems addresses boot services how a boot program initializes hardware and boots a client program Benefits of standard interfaces Reduced OS porting effort and cost Enables to development of standard boot programs (firmware and hypervisors) Key areas State of machine when control is transferred to client (e.g. registers, MMU, state of interrupts) Device Discovery definition of device tree Multi-cpu boot architecture ELF (Executable and Linking Format) for client programs Loosely related to IEEE 1275, and draws heavily on on-going work being done in the PowerPC Linux community (booting-without-of.txt) Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

43 Initial Machine State epapr defines the state of the hardware when control is transferred to a client program Registers MMU CPUs Memory State of I/O devices no interrupts or DMA Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

44 Initial State of Registers Register MSR R3 R4, R5, R8, R9 R7 TCR Value PR=0 supervisor state EE=0 interrupts disabled ME=0 machine check interrupt disabled IP=0 interrupt prefix-- low memory IR=0,DR=0 real mode (see note 1) IS=0,DS=0 address space 0 (see note 1) SF=0, CM=0, ICM=0 32-bit mode Effective address of the device tree image. 0 shall be the size of the boot or secondary IMA in bytes WRC=0, no watchdog timer reset will occur Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

45 Initial Mapped Areas (IMA) A client program s IMA is a region of memory that contains the entry points for a client program. Requirements: An IMA shall be virtually and physically contiguous An IMA shall start at effective address zero (0) which shall be mapped to a naturally aligned physical address The mapping shall not be invalidated except by a client program s explicit action The Translation ID (TID) field in the TLB entry shall be zero. The memory and cache access attributes (WIMGE) have the following requirements: WIMG unspecified E=0 (i.e., big-endian) An IMA may be mapped by a TLB entry larger than the IMA size, provided the MMU guarded attribute is set (G=1) An IMA may span multiple TLB entries. Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

46 Device Tree Overview A device tree is a tree data structure with nodes that describes the physical devices in a system Abstracts most hardware details out of the OS/client-- enables firmware to provide an OS with a complete description of the physical hardware in a system devices, hardware address map, interrupt routing Previously OSes were required to have hardcoded information about system hardware. Provides a basis for booting an operating system under a hypervisor in a partitioned system Each device node has property/value pairs that describe the device All nodes have a binding which document required properties the epapr documents bindings for some common devices Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

47 Examples / cpus cpu@0 soc compatible = simple-bus"; #address-cells = <1>; #size-cells = <1>; ranges = <0 e >; reg = <e >; cpu@1 soc ethernet serial compatible = "ns16550" reg = < > clock-frequency = <0> interrupts = <a 8> interrupt-parent = < &ipic > serial Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

48 epapr defines requirements for: Device Tree details Logical structure of the device tree node names, paths, properties Standard properties Hierarchy & routing of interrupts (including cascaded interrupt controllers) Representation of CPUs Memory Caches Device bindings for PCI Open PIC and ISA interrupt controllers Serial devcies Network devices Device Control Registers (DCR) Binary format of device tree DTB DTS syntax Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

49 Standard Properties compatible model phandle status #address-cells and #size-cells reg virtual-reg ranges dma-ranges interrupts interrupt-parent Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

50 UART Node Definition Example { device_type = "serial"; compatible = "fsl,ns16550", "ns16550"; reg = <11c >; clock-frequency = <0>; interrupts = <24 2>; interrupt-parent = <&mpic>; }; Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

51 Device Tree Compiler The typical process of using a device tree in an embedded Linux system is: An ASCII representation of the device tree is created in an a 'device tree source' (DTS) file DTS is compiled into a binary 'device tree blob' (DTB) file using a device tree compiler (DTC) tool DTB format is specified in the epapr Firmware loads the DTB into RAM and passes a pointer to the DTB to the Operating System kernel when it is started Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

52 Multicore boot architecture epapr describes specifics on how secondary CPUs are booted for a system with multiple CPUs Default boot architecture The boot program releases all CPUs from hardware reset 1 CPU is designated to be the client program s boot CPU All other CPUs are secondary and are placed into loop where the CPUs spin, waiting for a spin table field to change that directs them where to go Control is transferred to the client program on the boot CPU When the client program is ready for secondary cores to start, it releases them by writing the spin table field with the desired address The architecture allows for other custom-defined secondary CPU release mechanisms as well Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

53 Device Tree and Virtualization/Partitioning Each OS in a partitioned system is presented with a device tree describing that partition s subset of physical resources CPU cores Memory I/O devices Shared or virtualized resources would also be described Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

54 Porting an OS to the Freescale Embedded Hypervisor Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

55 Porting Overview Initial State and Boot CPU SOC Platform Additional Hypervisor-provided Resources Services Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

56 Initial State and Boot An OS must be minimally device tree aware Interrupt numbers passed in the device tree are handles used in VMPIC API hcalls In order to boot multiple CPUs the spin table release mechanism must be used Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

57 Hypervisor Device Trees HW, HV, Guest Guest #1 Guest #2 Guest Dev tree HW Dev tree hypervisor u-boot Guest Dev tree HV config tree dynamically created and loaded into guest memory Loaded into hypervisor memory by u- boot. Hardware dev tree /chosen node points to HV config tree Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

58 CPU Considerations The virtual CPU as seen by an OS behaves as a normal OS would expect, but there are some differences. Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

59 SOC Platform Devices The following devices are hypervisor-owned and should not be accessed directly: Interrupt controller (see vmpic services) The MPIC timers are not available for guest use Global Utilities Power management Clock control Reset control Peripheral Access Management Unit (PAMU) DDR memory controllers CPC (Platform cache) CCS (LAWs, coherence domains) Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

60 Hypervisor Services Interrupt controller (MPIC) Byte-channels character I/O stream Inter-partition signaling doorbell Partition management Start/stop/image-loading Partition management interrupts Reset Power management change clock frequency, power states Error Management Future CPU hot plug/unplug GPIO supports partitioning of GPIO pins IOMMU supports create/destroy mappings Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

61 Hypervisor Boot Flow Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

62 Hypervisor Device Trees HW, HV, Guest Guest #1 Guest #2 Guest Dev tree HW Dev tree hypervisor u-boot Guest Dev tree HV config tree dynamically created and loaded into guest memory Loaded into hypervisor memory by u- boot. Hardware dev tree /chosen node points to HV config tree Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

63 U-boot initialization Configuration of physical memory map (LAWs) Probing/initialization of DDR Enabling all caches, including platform cache Configuration clocks Setting up LIODNs Sets up epapr spin table for secondary CPUs, releases secondary CPUs Loads hypervisor image into memory Updates hardware device tree, loads it into memory Transfers control to the hypervisor U-boot Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

64 Hypervisor Hypervisor initialization PAMU initialization Coherence domain setup set up LAWS, CSDIDs Error Configuration e.g. single bit ECC error thresholds Driver initialization DDR CPC CCM UART Release secondary CPUs Partition instantiation/creation The boot CPU for each partition takes care of partition creation Control transfer to guest OS Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

65 Hypervisor Relocation Boot time view Runtime view 0x0 0x0 HV OS HV Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink

66 Summary A common usage model for multicore systems will be to run multiple operating systems on a single processor this requires partitioning. A hypervisor provides a good solution to enforce partition boundaries and provide services to manage global resources (like the interrupt controller). The Freescale Embedded Hypervisor in conjunction with hardware features of the QorIQ P4080 provides an efficient solution for partitioning secure partitions with minimal overhead. Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, CoreNet, the Energy Efficient Solutions logo, Flexis, MXC, Platform in a Package, Processor Expert, QorIQ, QUICC Engine, SMAROS, TurboLink 66

67