Dynamic VM Monitoring using Hypervisor Probes

Transcription

1 Dynamic VM Monitoring using Hypervisor Probes Z. J. Estrada, C. Pham, F. Deng, L. Yan, Z. Kalbarczyk, R. K. Iyer European Dependable Computing Conference

2 Dynamic VM Monitoring Goal On-demand VM Monitoring to reduce the effort required to harden computing systems against failures and attacks. Uptime requirements Effort required QA concerns Lack of knowledge 2

3 VM Monitoring Reliability & Security Monitoring Recording and analyzing a computer system to detect failures and attacks. Passive - polling based Active - event based 3

4 VM Monitoring 4

5 VM Monitoring 4

6 VM Monitoring 4

7 VM Monitor Monitor is running inside the hypervisor 5

8 VM Monitor VM execution reaches a hook 5

9 VM Monitor Control is transferred to the monitor 5

10 VM Monitor The monitor performs its monitoring function 5

11 VM Monitor Control is transferred back to the VM 5

12 VM Monitor The VM resumes normal execution 5

13 Hook-Based VM Monitoring Previous techniques: + Active monitoring + Protected hooks Guest OS only - no userspace Not dynamic - boot time config Require guest OS modifications 6

14 Goals Hook-based monitoring should: + be protected from attacks in the VM + be simple to use + not require guest OS modification + be runtime adaptable + allow for arbitrary hook placement 7

15 Hypervisor Probes 8

16 Hardware Assisted Virt. Host Mode (root) Guest Mode (non-root) User Kernel VMEntry VMExit User Kernel 9

17 Hypervisor Probes Event on guest execution Event transfers control to hypervisor (VM Exit) Perform monitoring after that event Hooks added/removed at runtime Monitors applications and the guest OS 10

18 Hprobe Architecture Status Checker Host System Hprobe user agent Probe Probe VM Probe Set/Remove probes Detector 1 Detector 2 Detector n ioctl( ) Hprobe Kernel agent Insert/Remove probes Set single step Host Linux kernel Event Forwarder Helper APIs KVM Hypervisor 11

19 Hprobes API int HPROBE_add_probe( ); int HPROBE_remove_probe( ); addr info: gva+cr3 vmid: unique id for VM vcpu type: vcpu state 12

20 Probe Event Forwarder VM Hypervisor... pushl %eax incl %eax decl %ebx... 13

21 Probe Event Forwarder VM Hypervisor... pushl %eax int3 decl %ebx... 13

22 Probe Event Forwarder... pushl %eax int3 decl %ebx... VM probe hit (int3) Hypervisor handler() Detector 13

23 Probe Event Forwarder VM probe hit (int3) Hypervisor handler() Reset inst.... pushl %eax incl %eax decl %ebx... 13

24 Probe Event Forwarder... pushl %eax incl %eax decl %ebx... VM probe hit (int3) execute inst. Hypervisor handler() Reset inst. single step 13

25 Probe Event Forwarder... pushl %eax int3 decl %ebx... VM probe hit (int3) execute inst. trap Hypervisor handler() Reset inst. single step rewrite int3 13

26 Probe Event Forwarder... pushl %eax int3 decl %ebx... VM probe hit (int3) execute inst. trap Hypervisor handler() Reset inst. single step rewrite int3... resume 13

27 Userspace Probe Challenge Guest Page Tables 14

28 Userspace Probe Challenge Guest Page Tables 14

29 Userspace Probe Challenge Guest Page Tables 14

30 Extended Page Tables (EPT) [1] Guest OS has full control over PTs 2nd set of HW PTs for GPA HPA Use EPT to write-protect Guest Page Table [1] 4/VT roadmap d Nakajima.pdf 15

31 Goals Hook-based monitoring should: + be protected from attacks in the VM + be simple to use + not require guest OS modification + be runtime adaptable + allow for arbitrary hook placement 16

32 Goals Hook-based monitoring should: be protected from attacks in the VM + be simple to use + not require guest OS modification + be runtime adaptable + allow for arbitrary hook placement 16

33 Goals Hook-based monitoring should: be protected from attacks in the VM be simple to use + not require guest OS modification + be runtime adaptable + allow for arbitrary hook placement 16

34 Goals Hook-based monitoring should: be protected from attacks in the VM be simple to use not require guest OS modification + be runtime adaptable + allow for arbitrary hook placement 16

35 Goals Hook-based monitoring should: be protected from attacks in the VM be simple to use not require guest OS modification be runtime adaptable + allow for arbitrary hook placement 16

36 Goals Hook-based monitoring should: be protected from attacks in the VM be simple to use not require guest OS modification be runtime adaptable allow for arbitrary hook placement 16

37 Hprobe Microbenchmarks noop kernel function execute 1M times VM kernel insert probe Hypervisor kernel start/stop hypercall record time user user [2] [2] Adapted from an image by Fei Deng 17

38 4.5 Hprobe Single Probe Latency 4.0 Time (µs) GHz E5430 Harpertown (2007) GHz E Sandy Bridge (2012) 18

39 Hook-based VM Monitoring Name Latency User Dynamic Modifications Lares 28µs No No Hypervisor/Guest SIM 0.40µs No No Hypervisor/Guest hprobes 2.6µs Yes Yes Hypervisor 19

40 Hook-based VM Monitoring Name Latency User Dynamic Modifications Lares 28µs No No Hypervisor/Guest SIM 0.40µs No No Hypervisor/Guest hprobes 2.6µs Yes Yes Hypervisor as-a-service is worth slight performance cost 19

41 Detectors What detectors can we build with hprobes? 20

42 Detectors What detectors can we build with hprobes? Arbitrarily chose events On-demand Access to VM memory & CPU state 20

43 Heartbeat/watchdog App Detector 21

44 Heartbeat/watchdog App Detector Insert Probe 21

45 Heartbeat/watchdog App Detector Insert Probe Probe Hit 21

46 Heartbeat/watchdog App Detector Insert Probe Probe Hit reset timer 21

47 Heartbeat/watchdog App Detector Insert Probe Probe Hit Probe Hit reset timer 21

48 Heartbeat/watchdog App Detector Insert Probe Probe Hit Probe Hit reset timer reset timer 21

49 Heartbeat/watchdog App Insert Probe Probe Hit Probe Hit Detector reset timer reset timer 21

50 Heartbeat/watchdog App Insert Probe Probe Hit Probe Hit Detector reset timer reset timer timer expires declare failure 21

51 Watchdog - Performance PI-QMC Main Loop Runtime No Detector With Detector Time (msec) x 2x 3x 4x 5x 6x 7x 8x 9x Internal Sample Loop Size 22

52 Detectors Infinite Loop Detector 23

53 Detectors Infinite Loop Detector Kernel or App-level Previously determined threshold Or register 23

54 Infinite Loop Detector for(i=0; i<10; i++) {... } //after loop 24

55 Infinite Loop Detector 1 st Probe (counter) for(i=0; i<10; i++) {... } //after loop 24

56 Infinite Loop Detector 1 st Probe (counter) 2 nd Probe (reset) for(i=0; i<10; i++) {... } //after loop 24

57 Without Infinite Loop Application Time (s) 95% CI (s) % overhead Normal N/A Naïve ILD - Page Naïve ILD - No Page Smart ILD - Page Smart ILD - No Page

58 Consider this situation 26

59 A vulnerability is announced 26

60 At a later time, a patch is released 26

61 What can we do?? 26

62 ? We may have to follow a maintenance window 26

63 ? Even when the bug and patch are coreleased 26

64 STOPGAP To mitigate risk, we would like a stopgap 26

65 STOPGAP Solution Use an Hprobe-based Detector 26

66 Emergency Detector Should be... easier than a patch simpler than a patch less disruptive than a patch less risky than a patch 27

67 Emergency Detector CVE Privilege Escalation in vmsplice() [3] Integer overflow in a struct iovec argument Corrupts OS (kernel) stack Execute attack payload struct iovec { void *iov_base; size_t iov_len; }; [3] aeb/linux/hh/hh-12.html#ss

68 Emergency Detector Added to running guest OS Detects malicious value that causes overflow Two modes of operation Read-only mode: does not change anything Fix mode: malicious value benign value 29

69 Emergency Detector Probe at vmsplice() syscall Get value of iov len off of the stack 30

70 Emergency Detector procedure VMSPLICE HANDLER(vcpu) iov pointer read guest(esp+arg offset) iov len read guest virt(iov pointer) if iov len BAD VALUE then HANDLE EXPLOIT ATTEMPT(vcpu) end if end procedure 31

71 Detector Performance Checkpoint/Restart In Userspace Two scientific computing applications Home Path-integral Quantum Monte Carlo Three cases: Normal: base case without monitoring hprobe: only monitor sys vmsplice Naïve: monitor all system calls 32

72 Detector Performance Application Runtime ± 95% CI (s) overhead (%) Normal ± F@H w/hprobe ± F@H w/naïve ± pi-qmc Normal ± pi-qmc w/hprobe ± pi-qmc w/naïve ±

73 Thoughts Zero overhead without vmsplice() Cloud provider doesn t need tenant to update Can be used while official fix is in QA Don t need full understanding of bug 34

74 VM Monitoring Techniques Lares (SP 08) SIM (CCS 09) HyperTap (DSN '14) Hprobes (EDCC'15) LibVMI (ACSAC 07) LiveWire (NDSS 03) Osck (ASPLOS 11) Antfarm (ATC 06) Lycosid (VEE 08) Virtuoso (SP 11) VMST (SP 12) On-demand Add/Remove Changes to VM Userspace Monitoring Root-of-trust (invariant) OS HW OS OS OS HW OS OS OS Active/Passive Mon. A (Hook) A P P A (Hook) A P P P Auto-generate Monitoring This Presentation Desirable Feature = Supported Feature = Unsupported Feature 35

75 Acknowledgements Collaborators: Cuong Pham, Fei Deng, Dr. Lok Yan, Prof. Zbigniew Kalbarczyk, Prof. Ravi Iyer 36

76 Summary VM Monitoring How hprobes work Microbenchmarks Emergency Detector 37