The Quest-V Separation Kernel

Transcription

1 The Quest-V Separation Kernel Richard West Computer Science

2 Goals Develop system for high-confidence (embedded) systems Mixed criticalities (timeliness and safety) Predictable real-time support Resistant to component failures & malicious manipulation Self-healing Online recovery of software component failures 2

3 Target Applications Healthcare Avionics Automotive Factory automation Robotics Space exploration Other safety-critical domains 3

4 Case Studies $327 million Mars Climate Orbiter Loss of spacecraft due to Imperial / Metric conversion error (September 23, 1999) 10 yrs & $7 billion to develop Ariane 5 rocket June 4, 1996 rocket destroyed during flight Conversion error from 64-bit double to 16-bit value 50+ million people in 8 states & Canada in 2003 without electricity due to software race condition 4

5 Approach Quest-V for multi-/many-core processors Distributed system on a chip Time as a first-class resource Cycle-accurate time accountability Separate sandbox kernels for system components Memory isolation using h/w-assisted memory virtualization Extended page tables (EPTs Intel) Nested page tables (NPTs AMD) Also need CPU, I/O, cache isolation, etc (later!) 6

6 Related Work Existing virtualized solutions for resource partitioning Wind River Hypervisor, XtratuM, PikeOS, Mentor Graphics Hypervisor Xen, Oracle PDOMs, IBM LPARs Muen, (Siemens) Jailhouse 7

7 Problem Traditional Virtual Machine approaches too expensive Require traps to VMM (a.k.a. hypervisor) to mux & manage machine resources for multiple guests e.g., ~1500 clock cycles VM-Enter/Exit on Xeon E5506 8

8 Traditional Approach (Type 1 VMM) VM VM VM VM... VM Type 1 VMM / Hypervisor Hardware (CPUs, memory, devices) 9

9 Contributions Quest-V Separation Kernel [WMC'13, VEE'14] Uses H/W virtualization to partition resources amongst services of different criticalities Each partition, or sandbox, manages its own CPU cores, memory area, and I/O devices w/o hypervisor intervention Hypervisor typically only needed for bootstrapping system + managing comms channels b/w sandboxes 10

10 Contributions Quest-V Separation Kernel Eliminates hypervisor intervention during normal virtual machine operations 11

11 Architecture Overview 12

12 Memory Partitioning Guest kernel page tables for GVA-to-GPA translation EPTs (a.k.a. shadow page tables) for GPA-toHPA translation EPTs modifiable only by monitors Intel VT-x: 1GB address spaces require 12KB EPTs w/ 2MB superpaging 13

13 Quest-V Linux Memory Layout 14

14 Quest-V Memory Partitioning 15

15 Memory Virtualization Costs Example Data TLB overheads Xeon E GHz, 4GB RAM 16

16 I/O Partitioning Device interrupts directed to each sandbox Use I/O APIC redirection tables Eliminates monitor from control path EPTs prevent unauthorized updates to I/O APIC memory area by guest kernels Port-addressed devices use in/out instructions VMCS configured to cause monitor trap for specific port addresses Monitor maintains device "blacklist" for each sandbox DeviceID + VendorID of restricted PCI devices 17

17 Quest-V I/O Partitioning Data Port: 0xCFC Address Port: 0xCF8 18

18 Monitor Intervention During normal operation only one monitor trap every 3-5 mins by CPUID No I/O Partitioning Exception (TF) I/O Partitioning (Block COM and NIC) VMCALL 2 2 I/O Instruction EPT Violation XSETBV 1 1 CPUID Table: Monitor Trap Count During Linux Sandbox Initialization 19

19 CPU Partitioning Scheduling local to each sandbox partitioned rather than global avoids monitor intervention Uses real-time VCPU approach for Quest native kernels [RTAS'11] 20

20 Predictability VCPUs for budgeted real-time execution of threads and system events (e.g., interrupts) Threads mapped to VCPUs VCPUs mapped to physical cores Sandbox kernels perform local scheduling on assigned cores Avoid VM-Exits to Monitor eliminate cache/tlb flushes 21

21 VCPUs in Quest(-V) Address Space Threads Main VCPUs I/O VCPUs PCPUs (Cores) 22

22 VCPUs in Quest(-V) Two classes Main for conventional tasks I/O for I/O event threads (e.g., ISRs) Scheduling policies Main sporadic server (SS) I/O priority inheritance bandwidthpreserving server (PIBS) 23

23 SS Scheduling Model periodic tasks Each SS has a pair (C,T) s.t. a server is guaranteed C CPU cycles every period of T cycles when runnable Guarantee applied at foreground priority background priority when budget depleted Rate-Monotonic Scheduling theory applies 24

24 PIBS Scheduling IO VCPUs have utilization factor, UV,IO IO VCPUs inherit priorities of tasks (or Main VCPUs) associated with IO events Currently, priorities are ƒ(t) for corresponding Main VCPU IO VCPU budget is limited to: TV,main* UV,IO for period TV,main 25

25 PIBS Scheduling IO VCPUs have eligibility times, when they can execute te = t + Cactual / UV,IO t = start of latest execution t >= previous eligibility time 26

26 Example VCPU Schedule 27

27 Sporadic Constraint Worst-case preemption by a sporadic task for all other tasks is not greater than that caused by an equivalent periodic task (1) Replenishment, R must be deferred at least t+t V (2) Can be deferred longer (3) Can merge two overlapping replenishments R1.time + R1.amount >= R2.time then MERGE Allow replenishment of R1.amount +R2.amount at R1.time 28

28 Example Replenishments amount, time VCPU 1 (C=20, T=50, Start=0) VCPU 0 (C=10, T=40, Start=1) 20,00 00,00 00,00 (A) (B) ,00 18,50 00, ,40 18,50 00, Replenishment Queue Element 18,50 02,90 00, ,50 02,90 16, ,80 02,90 16, IOVCPU (Utilization=4%) 02,90 16,100 16,100 02,130 02,130 02, Corrected Algorithm Premature Replenishment Interval [t=0,100] (A) VCPU 1 = 40%, (B) VCPU 1 = 46% 29

29 Utilization Bound Test Sandbox with 1 PCPU, n Main VCPUs, and m I/O VCPUs Ci = Budget Capacity of Vi Ti = Replenishment Period of Vi Main VCPU, Vi Uj = Utilization factor for I/O VCPU, Vj n 1 Ci + Ti i=0 m 1 n (2 Uj) Uj n ( 2 1) j=0 30

30 Cache Partitioning Shared caches controlled using color-aware memory allocator Cache occupancy prediction based on h/w performance counters E' = E + (1-E/C) * ml E/C * mo Enhanced with hits + misses [Book Chapter, OSR'11, PACT'10] 31

31 Linux Front End For low criticality legacy services Based on Puppy Linux Runs entirely out of RAM including root filesystem Low-cost paravirtualization less than 100 lines Restrict observable memory Adjust DMA offsets Grant access to VGA framebuffer + GPU Quest native SBs tunnel terminal I/O to Linux via shared memory using special drivers 32

32 Quest-V Linux Screenshot 33

33 Quest-V Linux Screenshot 1 CPU MB No VMX or EPT flags 34

34 Quest-V Performance Overhead Measured time to play back 1080P MPEG2 video from the x264 HD video benchmark Mini-ITX Intel Core i5-2500k 4-core, HD3000 graphics, 4GB RAM mplayer Benchmark 35

35 Conclusions Quest-V separation kernel built from scratch Distributed system on a chip Uses (optional) h/w virtualization to partition resources into sandboxes Protected comms channels b/w sandboxes Sandboxes can have different criticalities Linux front-end for less critical legacy services Sandboxes responsible for local resource management avoids monitor involvement 36

36 Quest-V Status About 11,000 lines of kernel code 200,000+ lines including lwip, drivers, regression tests SMP, IA32, paging, VCPU scheduling, USB, PCI, networking, etc Quest-V requires BSP to send INIT-SIPI-SIPI to APs, as in SMP system BSP launches 1st (guest) sandbox APs VM fork their sandboxes from BSP copy 37

37 Future Work Online fault detection and recovery Technologies for secure monitors e.g., Intel TXT + VT-d Separation kernel support for: Accelerators / GPUs (time partitioning) NoCs Heterogeneous platforms (ala Helios satellite kernels) See for more details 38

38 Quest-V Demo Bootstrapping Quest native kernel (core 0) + Linux (core 1) Linux kernel + filesystem in RAM Secure comms channel b/w Quest SB & Linux SB using a pseudo-char device /dev/qsbx device for each sandbox x Triple modular redundancy (TMR) fault recovery for unmanned aerial vehicle (UAV) 39

39 The Quest Team Richard West Ye Li Eric Missimer Matt Danish Gary Wong 40

40 Other (Current) Developments Port of Quest to Intel Galileo Arduino Quest RT-USB host controller stack [RTAS'13] 41

41 10+ Years On... Intelligent transportation systems V2V and V2I communications Driverless cars (e.g., Google, CMU, Stanford, Oxford RobotCar, etc) Humanoid robots Complex sensing + processing networks 42