Distributed System Monitoring and Failure Diagnosis using Cooperative Virtual Backdoors

Transcription

1 Distributed System Monitoring and Failure Diagnosis using Cooperative Virtual Backdoors Benoit Boissinot E.N.S Lyon directed by Christine Morin IRISA/INRIA Rennes Liviu Iftode Rutgers University Phenix Workshop December 2006

2 Outline 1 Introduction 2 Background 3 Virtual Backdoors 4 Distributed Virtual Backdoors 5 Related Work 6 Conclusion

3 Outline 1 Introduction 2 Background 3 Virtual Backdoors 4 Distributed Virtual Backdoors 5 Related Work 6 Conclusion

4 Robust Computing Achieving Robustness better programming execution monitoring

5 Robust Computing Achieving Robustness better programming execution monitoring Monitoring failures detection attacks detection management load balancing

6 Monitoring Principles Monitor Properties Non-intrusive: no need to modify the target OS

7 Monitoring Principles Monitor Properties Non-intrusive: no need to modify the target OS Tamper-proof: no possible intrusion

8 Monitoring Principles Monitor Properties Non-intrusive: no need to modify the target OS Tamper-proof: no possible intrusion Autonomous: no involvement of the target OS

9 Monitoring Principles Monitor Properties Non-intrusive: no need to modify the target OS Tamper-proof: no possible intrusion Autonomous: no involvement of the target OS Consistent view of the OS state

10 Monitoring Principles Monitor Properties Non-intrusive: no need to modify the target OS Tamper-proof: no possible intrusion Autonomous: no involvement of the target OS Consistent view of the OS state Programmable: for flexibility

11 Monitoring Principles Monitor Properties Non-intrusive: no need to modify the target OS Tamper-proof: no possible intrusion Autonomous: no involvement of the target OS Consistent view of the OS state Programmable: for flexibility Failsafe communications: reliable inter-monitor communications for distributed monitoring

12 Monitoring Principles (Autonomy) Two ways to obtain autonomy:

13 Monitoring Principles (Autonomy) Two ways to obtain autonomy: Hardware Isolation External programmable devices with access to the memory:

14 Monitoring Principles (Autonomy) Two ways to obtain autonomy: Hardware Isolation External programmable devices with access to the memory: PCI devices, Firewire

15 Monitoring Principles (Autonomy) Two ways to obtain autonomy: Hardware Isolation External programmable devices with access to the memory: PCI devices, Firewire Software Isolation Separate path of execution:

16 Monitoring Principles (Autonomy) Two ways to obtain autonomy: Hardware Isolation External programmable devices with access to the memory: PCI devices, Firewire Software Isolation Separate path of execution: Virtualization technology

17 Outline 1 Introduction 2 Background 3 Virtual Backdoors 4 Distributed Virtual Backdoors 5 Related Work 6 Conclusion

18 Friendly Backdoor Backdoor A hidden software or hardware mechanism, usually created for testing and troubleshooting. Original Implementation American National Standard for Telecommunications RDMA network card sitting on a PCI bus no overhead on the target CPU only memory is accessible remotely synchronization with the target OS is hard

19 Virtual Machines Definitions Virtual Machine (VM): Virtualized computer system, functional equivalent of the real machine. Virtual Machine Monitor (VMM): Software providing a virtualized environment to VMs.

20 Virtual Machines Definitions Virtual Machine (VM): Virtualized computer system, functional equivalent of the real machine. Virtual Machine Monitor (VMM): Software providing a virtualized environment to VMs. Technologies used Type I VMM

21 Virtual Machines Definitions Virtual Machine (VM): Virtualized computer system, functional equivalent of the real machine. Virtual Machine Monitor (VMM): Software providing a virtualized environment to VMs. Technologies used Type I VMM Type II VMM

22 Virtual Machines Definitions Virtual Machine (VM): Virtualized computer system, functional equivalent of the real machine. Virtual Machine Monitor (VMM): Software providing a virtualized environment to VMs. Technologies used Type I VMM Type II VMM x86 virtualization is not trivial:

23 Virtual Machines Definitions Virtual Machine (VM): Virtualized computer system, functional equivalent of the real machine. Virtual Machine Monitor (VMM): Software providing a virtualized environment to VMs. Technologies used Type I VMM Type II VMM x86 virtualization is not trivial: Emulation: VmWare

24 Virtual Machines Definitions Virtual Machine (VM): Virtualized computer system, functional equivalent of the real machine. Virtual Machine Monitor (VMM): Software providing a virtualized environment to VMs. Technologies used Type I VMM Type II VMM x86 virtualization is not trivial: Emulation: VmWare Paravirtualization: Xen

25 Virtual Machines Definitions Virtual Machine (VM): Virtualized computer system, functional equivalent of the real machine. Virtual Machine Monitor (VMM): Software providing a virtualized environment to VMs. Technologies used Type I VMM Type II VMM x86 virtualization is not trivial: Emulation: VmWare Paravirtualization: Xen New instruction set: Intel, AMD

26 Xen Virtual Machine Monitor Developed at the University of Cambridge. Xen VMM Open source technology

27 Xen Virtual Machine Monitor Developed at the University of Cambridge. Xen VMM Open source technology Paravirtualized

28 Xen Virtual Machine Monitor Developed at the University of Cambridge. Xen VMM Open source technology Paravirtualized Thin VMM, virtualization in a privileged VM (dom0)

29 Xen Virtual Machine Monitor Developed at the University of Cambridge. Xen VMM Open source technology Paravirtualized Thin VMM, virtualization in a privileged VM (dom0) Multiples OS s ported: Linux, *BSD, Minix

30 Xen Virtual Machine Monitor Developed at the University of Cambridge. Xen VMM Open source technology Paravirtualized Thin VMM, virtualization in a privileged VM (dom0) Multiples OS s ported: Linux, *BSD, Minix Hardware VT support

31 Outline 1 Introduction 2 Background 3 Virtual Backdoors 4 Distributed Virtual Backdoors 5 Related Work 6 Conclusion

32 Platform Choice Why a VM based backdoor? Run on common hardware Separate path of execution for each VM Control the VM Why Xen VMM? Thin means secure Fast Open source

33 Virtual Backdoor Design Where? VMM What? dedicated, privileged VM read/write a guest operating system memory extract state information from the guest operating system consistent view

34 Backdoor Programming Unix philosophy: everything is file

35 Backdoor Programming Unix philosophy: everything is file /dev/kmem: kernel virtual memory is a file

36 Backdoor Programming Unix philosophy: everything is file /dev/kmem: kernel virtual memory is a file /dev/domain mem: virtual memory of other VM target VM is selectable via an ioctl

37 Example

38 Memory Access Implementation Foreign Pages Mapping How to access a memory location from a remote VM? The VMM provides an interface to map a foreign page into the privileged VM address space. But we need the physical page frame number.

39 Memory Translation SoftMMU Software virtual address translation Page table walking Three pages mapping No swapping for the target kernel

40 Discovery of OS State How to make sense of the remote OS memory? We need to know the layout and the of each OS data structure Modified ELF parser to extract the debug informations from an object file Library to simplify the access to this information

41 Validation Experiment Process list walking Walk the linked list of the processes structures Output informations for each process Compare the output with ps

42 Outline 1 Introduction 2 Background 3 Virtual Backdoors 4 Distributed Virtual Backdoors 5 Related Work 6 Conclusion

43 Distributed Operating System Monitoring Single System Image Operating System Operating system running on multiples computer Provide a view of a unique operating system Need fault-tolerance, operating system consistency checks Example: Kerrighed

44 Distributed Monitoring: Cooperative Backdoors Network Block Device (nbd) Small modifications Server: export # of requests processed Client: export # of requests sent Cooperative Backdoor Backdoors retrieve the data exported Use the XML-RPC protocol One backdoor aggregate the data Record the timestamps when the # change Compute the end-to-end delay

45 Outline 1 Introduction 2 Background 3 Virtual Backdoors 4 Distributed Virtual Backdoors 5 Related Work 6 Conclusion

46 PCI based Backdoor Backdoor Architecture Work from the DiscoLab team Based on Myrinet networking cards with RDMA Access only to the memory Communicate with high-speed private network

47 Virtual Machine Based Intrusion Detection VMM Based Monitoring Plato/Revirt from the CoVirt group (U. of Michigan) Based on UMLinux (type II VMM), high virtualization overhead Interposition / Logging / Replay / Checkpoint Used for intrusion detection

48 Outline 1 Introduction 2 Background 3 Virtual Backdoors 4 Distributed Virtual Backdoors 5 Related Work 6 Conclusion

49 Contributions Propose a virtual backdoor architecture for remote monitoring Virtual backdoor designed, implemented and tested over Xen VMM Show how virtual backdoors can cooperate to monitor distributed state

50 Future Work Cache virtual addresses lookup Synchronization with the target OS Apply with a distributed operating systems

51 The End Thank you!