Distributed System Monitoring and Failure Diagnosis using Cooperative Virtual Backdoors

Distributed System Monitoring and Failure Diagnosis using Cooperative Virtual Backdoors Benoit Boissinot E.N.S Lyon directed by Christine Morin IRISA/INRIA Rennes Liviu Iftode Rutgers University Phenix Workshop December 2006

Outline 1 Introduction 2 Background 3 Virtual Backdoors 4 Distributed Virtual Backdoors 5 Related Work 6 Conclusion

Robust Computing Achieving Robustness better programming execution monitoring

Robust Computing Achieving Robustness better programming execution monitoring Monitoring failures detection attacks detection management load balancing

Monitoring Principles Monitor Properties Non-intrusive: no need to modify the target OS

Monitoring Principles Monitor Properties Non-intrusive: no need to modify the target OS Tamper-proof: no possible intrusion

Monitoring Principles Monitor Properties Non-intrusive: no need to modify the target OS Tamper-proof: no possible intrusion Autonomous: no involvement of the target OS

Monitoring Principles Monitor Properties Non-intrusive: no need to modify the target OS Tamper-proof: no possible intrusion Autonomous: no involvement of the target OS Consistent view of the OS state

Monitoring Principles Monitor Properties Non-intrusive: no need to modify the target OS Tamper-proof: no possible intrusion Autonomous: no involvement of the target OS Consistent view of the OS state Programmable: for flexibility

Monitoring Principles Monitor Properties Non-intrusive: no need to modify the target OS Tamper-proof: no possible intrusion Autonomous: no involvement of the target OS Consistent view of the OS state Programmable: for flexibility Failsafe communications: reliable inter-monitor communications for distributed monitoring

Monitoring Principles (Autonomy) Two ways to obtain autonomy:

Monitoring Principles (Autonomy) Two ways to obtain autonomy: Hardware Isolation External programmable devices with access to the memory:

Monitoring Principles (Autonomy) Two ways to obtain autonomy: Hardware Isolation External programmable devices with access to the memory: PCI devices, Firewire

Monitoring Principles (Autonomy) Two ways to obtain autonomy: Hardware Isolation External programmable devices with access to the memory: PCI devices, Firewire Software Isolation Separate path of execution:

Monitoring Principles (Autonomy) Two ways to obtain autonomy: Hardware Isolation External programmable devices with access to the memory: PCI devices, Firewire Software Isolation Separate path of execution: Virtualization technology

Friendly Backdoor Backdoor A hidden software or hardware mechanism, usually created for testing and troubleshooting. Original Implementation American National Standard for Telecommunications RDMA network card sitting on a PCI bus no overhead on the target CPU only memory is accessible remotely synchronization with the target OS is hard

Virtual Machines Definitions Virtual Machine (VM): Virtualized computer system, functional equivalent of the real machine. Virtual Machine Monitor (VMM): Software providing a virtualized environment to VMs.

Virtual Machines Definitions Virtual Machine (VM): Virtualized computer system, functional equivalent of the real machine. Virtual Machine Monitor (VMM): Software providing a virtualized environment to VMs. Technologies used Type I VMM

Virtual Machines Definitions Virtual Machine (VM): Virtualized computer system, functional equivalent of the real machine. Virtual Machine Monitor (VMM): Software providing a virtualized environment to VMs. Technologies used Type I VMM Type II VMM

Virtual Machines Definitions Virtual Machine (VM): Virtualized computer system, functional equivalent of the real machine. Virtual Machine Monitor (VMM): Software providing a virtualized environment to VMs. Technologies used Type I VMM Type II VMM x86 virtualization is not trivial:

Virtual Machines Definitions Virtual Machine (VM): Virtualized computer system, functional equivalent of the real machine. Virtual Machine Monitor (VMM): Software providing a virtualized environment to VMs. Technologies used Type I VMM Type II VMM x86 virtualization is not trivial: Emulation: VmWare

Virtual Machines Definitions Virtual Machine (VM): Virtualized computer system, functional equivalent of the real machine. Virtual Machine Monitor (VMM): Software providing a virtualized environment to VMs. Technologies used Type I VMM Type II VMM x86 virtualization is not trivial: Emulation: VmWare Paravirtualization: Xen

Virtual Machines Definitions Virtual Machine (VM): Virtualized computer system, functional equivalent of the real machine. Virtual Machine Monitor (VMM): Software providing a virtualized environment to VMs. Technologies used Type I VMM Type II VMM x86 virtualization is not trivial: Emulation: VmWare Paravirtualization: Xen New instruction set: Intel, AMD

Xen Virtual Machine Monitor Developed at the University of Cambridge. Xen VMM Open source technology

Xen Virtual Machine Monitor Developed at the University of Cambridge. Xen VMM Open source technology Paravirtualized

Xen Virtual Machine Monitor Developed at the University of Cambridge. Xen VMM Open source technology Paravirtualized Thin VMM, virtualization in a privileged VM (dom0)

Xen Virtual Machine Monitor Developed at the University of Cambridge. Xen VMM Open source technology Paravirtualized Thin VMM, virtualization in a privileged VM (dom0) Multiples OS s ported: Linux, *BSD, Minix

Xen Virtual Machine Monitor Developed at the University of Cambridge. Xen VMM Open source technology Paravirtualized Thin VMM, virtualization in a privileged VM (dom0) Multiples OS s ported: Linux, *BSD, Minix Hardware VT support

Platform Choice Why a VM based backdoor? Run on common hardware Separate path of execution for each VM Control the VM Why Xen VMM? Thin means secure Fast Open source

Virtual Backdoor Design Where? VMM What? dedicated, privileged VM read/write a guest operating system memory extract state information from the guest operating system consistent view

Backdoor Programming Unix philosophy: everything is file

Backdoor Programming Unix philosophy: everything is file /dev/kmem: kernel virtual memory is a file

Backdoor Programming Unix philosophy: everything is file /dev/kmem: kernel virtual memory is a file /dev/domain mem: virtual memory of other VM target VM is selectable via an ioctl

Example

Memory Access Implementation Foreign Pages Mapping How to access a memory location from a remote VM? The VMM provides an interface to map a foreign page into the privileged VM address space. But we need the physical page frame number.

Memory Translation SoftMMU Software virtual address translation Page table walking Three pages mapping No swapping for the target kernel

Discovery of OS State How to make sense of the remote OS memory? We need to know the layout and the of each OS data structure Modified ELF parser to extract the debug informations from an object file Library to simplify the access to this information

Validation Experiment Process list walking Walk the linked list of the processes structures Output informations for each process Compare the output with ps

Distributed Operating System Monitoring Single System Image Operating System Operating system running on multiples computer Provide a view of a unique operating system Need fault-tolerance, operating system consistency checks Example: Kerrighed

Distributed Monitoring: Cooperative Backdoors Network Block Device (nbd) Small modifications Server: export # of requests processed Client: export # of requests sent Cooperative Backdoor Backdoors retrieve the data exported Use the XML-RPC protocol One backdoor aggregate the data Record the timestamps when the # change Compute the end-to-end delay

PCI based Backdoor Backdoor Architecture Work from the DiscoLab team Based on Myrinet networking cards with RDMA Access only to the memory Communicate with high-speed private network

Virtual Machine Based Intrusion Detection VMM Based Monitoring Plato/Revirt from the CoVirt group (U. of Michigan) Based on UMLinux (type II VMM), high virtualization overhead Interposition / Logging / Replay / Checkpoint Used for intrusion detection

Contributions Propose a virtual backdoor architecture for remote monitoring Virtual backdoor designed, implemented and tested over Xen VMM Show how virtual backdoors can cooperate to monitor distributed state

Future Work Cache virtual addresses lookup Synchronization with the target OS Apply with a distributed operating systems

The End Thank you!