Machine Virtualization: Efficient Hypervisors, Stealthy Malware Muli Ben-Yehuda Technion & Hypervisor Technologies and Consulting Ltd Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 1 / 21
Background: x86 machine virtualization Running multiple different unmodified operating systems Each in an isolated virtual machine Simultaneously On the x86 architecture Many uses: live migration, record & replay, testing,..., security Foundation of IaaS cloud computing Used nearly everywhere Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 2 / 21
x86 virtualization primer How does it work? Popek and Goldberg s virtualization model [Popek74]: Trap and emulate Privileged instructions trap to the hypervisor Hypervisor emulates their behavior Without hardware support With hardware support Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 3 / 21
What is a rootkit? First you take control. How? Then you hide to avoid detection and maintain control. How? Usual methods are ugly and intrusive: easy to detect! Can rootkit authors do better? Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 4 / 21
Hypervisor-level rootkits Hypervisors have full control over the hardware Hypervisors can trap any operating system event Code can enter hypervisor-mode at any time Bluepill: run the rootkit as the hypervisor Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 5 / 21
Bluepill: a hypervisor level rootkit [Rutkowska06] Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 6 / 21
Recursive Bluepill Bluepill installs itself on the fly Bluepill is now the hypervisor Reminder: x86 only supports one hypervisor in hardware So how can you bluepill bluepill? Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 7 / 21
The Turtles project: Nested x86 Virtualization Efficient nested virtualization for Intel x86 based on KVM Runs multiple guest hypervisors and VMs The Turtles Project: Design and Implementation of Nested Virtualization, [Ben-Yehuda10] Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 8 / 21
What is the Turtles project? (cont ) Nested VMX virtualization for nested CPU virtualization Multi-dimensional paging for nested MMU virtualization Multi-level device assignment for nested I/O virtualization Micro-optimizations to make it go fast + Muli Ben-Yehuda (Technion & Hypervisor) + Efficient Hypervisors, Stealthy Malware = Cyberday, 2013 9 / 21
Theory of nested CPU virtualization Trap and emulate[popekgoldberg74] it s all about the traps Single-level (x86) vs. multi-level (e.g., z/vm) Single level one hypervisor, many guests Turtles approach: L 0 multiplexes the hardware between L 1 and L 2, running both as guests of L 0 without either being aware of it (Scheme generalized for n levels; Our focus is n=2) Guest L2 Guest L2 L1 Guest Hypervisor Guest Guest Hypervisor L1 Guest L2 Guest L2 Guest L0 Host Hypervisor L0 Host Hypervisor Hardware Hardware Multiple logical levels Multiplexed on a single level Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 10 / 21
Detecting hypervisor-based rootkits Bluepill authors claim undetectable Compatibility is Not Transparency: VMM Detection Myths and Realities [Garfinkel07] Hardware discrepancies Resource-sharing attacks Timing attacks: PCI register access, page-faults on MMIO access, cpuid timing vs. nops Can you trust time? Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 11 / 21
The Dual Role of a Hypervisor Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 12 / 21
Background: interrupts IDTR IDT IDT Entry Address IDT Entry Vector 1 Vector 2 Limit IDT Register IDT Entry Interrupt Descriptor Table Vector n Interrupt handlers I/O devices raise interrupts CPU temporarily stops the currently executing code CPU jumps to a pre-specified interrupt handler Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 13 / 21
Interrupts as an Attack Vector Follow the White Rabbit [Rutkowska11] Tell the device to generate interesting interrupts Attack: fool the CPU into SIPI Attack: syscall/hypercall injection In interrupt-based attacks an untrusted guest generates malicious interrupts which are handled in host mode Protect: handle interrupts in guest not host mode Serve: bare-metal performance! Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 14 / 21
ELI: Exitless Interrupts guest (a) Baseline Physical Interrupt Interrupt Injection Interrupt Completion hypervisor ELI (b) delivery Interrupt Completion guest hypervisor ELI delivery & (c) completion guest hypervisor (d) bare-metal (time) ELI: direct interrupts for unmodified, untrusted guests ELI: Bare-Metal Performance for I/O Virtualization, Gordon12 Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 15 / 21
ELI: delivery Shadow IDT Guest IDT Interrupt Handler IDTR Limit IDT Entry IDT Entry P=0 P=1 #NP Handler Assigned Interrupt Shadow IDT ELI Delivery VM Non-assigned Interrupt (#NP/#GP exit) IDT Entry P=0 #NP Hypervisor IDT Entry #GP Physical Interrupt All interrupts are delivered directly to the guest Host and other guests interrupts are bounced back to the host... without the guest being aware of it Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 16 / 21
ELI: signaling completion Guests signal interrupt completions by writing to the Local Advance Programmable Interrupt Controller (LAPIC) End-of-Interrupt (EOI) register Old LAPIC: hypervisor traps load/stores to LAPIC page x2apic: hypervisor can trap specific registers Signaling completion without trapping requires x2apic ELI gives the guest direct access only to the EOI register Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 17 / 21
ELI: threat model Threats: malicious guests might try to: keep interrupts disabled signal invalid completions consume other guests or host interrupts Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 18 / 21
ELI: protection VMX preemption timer to force exits instead of timer interrupts Ignore spurious EOIs Protect critical interrupts by: Delivering them to a non-eli core if available Redirecting them as NMIs unconditional exit Use IDTR limit to force #GP exits on critical interrupts Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 19 / 21
Conclusions Machine virtualization be used for good, or evil How do you protect and serve? Happy hacking! Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 20 / 21
Questions? muli@cs.technion.ac.il mulix@hypervisorconsulting.com Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 21 / 21